- 🛡️ Durability first: Elasticsearch writes each index and delete operation to a per-shard transaction log before acknowledging it, so recent writes can be replayed after a crash.
- ⚙️ Platform split: Elasticsearch currently documents a 10 GB flush threshold, while OpenSearch exposes different current controls and older 2.12 guidance used 512 MB.
- 🎯 Tuning rule: request durability is the safer production baseline; async durability trades lower fsync pressure for a defined window in which acknowledged writes can be lost after a failure.
- 🔬 Investigative finding: larger flush thresholds can improve indexing throughput, but recovery may have more operations to replay, so the effect needs workload-specific benchmarks.
- ☁️ Cloud direction: Elastic and OpenSearch are moving durability into object or remote storage without removing the need for an operation log.
- 📊 Reader decision: change a setting only after capturing flush, recovery, indexing, and uncommitted-operation baselines for that index.
Translogs are the per-shard operation logs that let Elasticsearch acknowledge writes without forcing an expensive Lucene commit after every document, and that durability shortcut becomes a recovery liability if the log grows faster than the cluster can safely replay it. In current Elasticsearch documentation, request durability remains the default, the sync interval is 5 seconds for asynchronous syncing, and the flush threshold is 10 GB, which is a very different baseline from older tuning advice many operators still quote (Elastic, n.d.-a).
The key is to separate three events that are often confused. A refresh makes newly indexed documents searchable. An fsync of the transaction log makes recent operations durable. A flush performs a Lucene commit and starts a new transaction-log generation. Treating those as the same operation leads to bad tuning decisions, especially during high-volume ingestion.
The distinction matters beyond classic search clusters. Retrieval systems increasingly depend on fresh indexes for RAG, research assistants, and citation-first answer engines. Our overview of citation-first search shows why index freshness and retrieval reliability sit directly under the user-facing answer layer. This guide focuses on the storage mechanics beneath that experience: what the transaction log protects, which settings are worth touching, what to measure first, and where Elasticsearch and OpenSearch now diverge.
Why the Transaction Log Exists Between Memory and Lucene
Lucene commits are durable but comparatively expensive, so Elasticsearch does not perform one for every index or delete request. Instead, a shard applies the operation to the internal Lucene index and appends it to the transaction log before the request is acknowledged. If the process, operating system, or machine fails before the next Lucene commit, recovery can replay acknowledged operations from that log (Elastic, n.d.-a).
Lucene segment files provide the committed state, while the transaction log bridges the gap between commits. A flush commits Lucene and opens a new log generation. Elasticsearch performs this automatically, and current API documentation says manual flush calls are rarely needed (Elastic, n.d.-b).
OpenSearch describes the same lifecycle: the log protects updates, refresh makes them searchable, and flush makes the Lucene state durable (OpenSearch Project, n.d.-a). The projects now expose different settings and storage paths, so tuning recipes should not be copied between them.
The Three Controls That Actually Matter
Durability changes the failure contract
Elasticsearch supports request and async durability. With request durability, success is reported only after the transaction log has been fsynced and committed on the primary and allocated replicas. With async durability, syncing happens on the configured interval, so acknowledged writes since the last sync can be discarded after a failure (Elastic, n.d.-a).
For data that cannot be regenerated, request durability is the safer baseline. Async can suit rebuildable telemetry, bulk imports, or derived indexes. The real question is whether the business can tolerate the recovery point objective created by the sync interval.
Sync interval controls the async loss window
The sync interval matters primarily when asynchronous durability is used. Elasticsearch documents a 5-second default and does not allow values below 100 ms. OpenSearch also documents a 5-second default and 100 ms minimum for index.translog.sync_interval (Elastic, n.d.-a; OpenSearch Project, n.d.-b).
Shortening the interval reduces the loss window but increases fsync frequency; lengthening it does the reverse. Storage type and node load determine the real cost, so benchmark the actual cluster instead of copying a number.
Flush threshold trades steady-state work for recovery work
Elasticsearch currently documents a 10 GB default for index.translog.flush_threshold_size and states that the log will not exceed 1 percent of disk size. Reaching the threshold triggers a flush and a new Lucene commit point (Elastic, n.d.-a).
OpenSearch needs separate treatment. Its current settings page lists sync, generation, retention, remote-store, and periodic-flush controls, while the captured latest page does not list the same flush-threshold entry. OpenSearch 2.12 guidance used 512 MB and suggested larger values for pure indexing while warning about longer recovery (OpenSearch Project, 2024). That advice is version-specific.
Elasticsearch and OpenSearch Are No Longer Identical
The mechanism remains familiar, but defaults and surrounding architecture are no longer interchangeable. Elastic is pushing stateless and serverless storage, while OpenSearch has developed remote-backed storage and segment replication.
Document the exact distribution and version before tuning. Older advice can persist long after defaults or exposed settings change. Version drift is now part of the problem.
| Area | Elasticsearch | OpenSearch | Operational reading |
| Write durability | request default; async optional | request and async supported; remote policy can restrict async | Choose by acceptable loss window |
| Sync interval | 5s default; 100ms minimum | 5s default; 100ms minimum | Relevant mainly when async durability is enabled |
| Flush threshold | 10 GB current documented default | Version-sensitive; 2.12 tuning guide documented 512 MB | Do not copy values between versions or platforms |
| Manual flush | Rarely needed | Use sparingly | Let automatic heuristics work unless maintenance requires otherwise |
| Remote durability | Serverless can upload log data to object storage | Remote-backed storage persists log data remotely | Durability is moving beyond local disk |
A Safe Production Tuning Workflow
Begin with a symptom, not a setting: high indexing latency, excessive flush time, slow recovery, disk pressure, or remote-store delay. Capture a baseline for the affected index. Elastic engineer Philipp Kahr likewise recommends realistic tests using your own logs and ingest pipelines (Kahr, 2025).
Then isolate one variable. Test async only on reproducible data, increase thresholds modestly, and include recovery in the success criteria. If recovery is already slow, reducing uncommitted work may matter more than higher steady-state throughput.
For search-backed AI applications, the index is part of a larger retrieval pipeline. The indexing layer should be benchmarked alongside chunking, embedding, retrieval, and reranking. Our guide to building a personal AI research assistant uses that same separation of stages because a fast model cannot compensate for a stale or unstable index.
GET /my-index/_stats/translog,flush,recovery,indexing
PUT /my-index/_settings
{
“index”: {
“translog.durability”: “request”,
“translog.sync_interval”: “5s”
}
}
What to Measure Before You Change Anything
Both platforms expose transaction-log and flush statistics through index stats APIs, alongside merge, recovery, and indexing metrics (Elastic, n.d.-c; OpenSearch Project, n.d.-c). Those measurements turn a tuning change into an experiment.
Capture metrics over a representative ingest window and repeat under the same load. For mixed workloads, include search latency and refresh behavior. More documents per second is not a win if restart recovery becomes unacceptable.
Server and crawler logs can also reveal whether slow indexing is actually upstream or downstream of the search engine. Our AI search visibility audit explains a broader log-first diagnostic approach: observe status codes, response times, access patterns, and technical failures before rewriting content or changing infrastructure. The same discipline applies here.
| Signal | What to capture | Why it matters | Warning pattern |
| Translog | operations, size, uncommitted ops and bytes | Shows outstanding recovery work | Uncommitted bytes keep climbing |
| Flush | flush count and total time | Reveals how often Lucene commits are occurring | Flush time rises after a threshold change |
| Recovery | recovered ops, duration and throttle time | Measures restart or movement cost | Recovery worsens after tuning |
| Indexing | throughput, p50/p95 latency, failures and throttling | Confirms whether the change actually helped ingestion | Average improves but tail latency or failures worsen |
| Disk and merge | disk utilization, segment count and merge time | Catches side effects outside the transaction log | Larger flushes shift pressure into merges or disk watermarks |
Recovery Problems: When the Log Becomes the Symptom
A large log matters when replay time, disk use, or shard recovery violates the service objective. Recovery restores committed Lucene state and then replays acknowledged operations outside the last commit. More uncommitted work means more recovery work.
Do not respond by forcing constant flushes. Both platforms already flush automatically, and their documentation treats manual flush as an administrative tool (Elastic, n.d.-b; OpenSearch Project, n.d.-d).
Instead, diagnose why uncommitted work accumulates. Check whether the ingest rate jumped, storage latency changed, the cluster is merge-bound, a node is near disk watermarks, or shard allocation is creating repeated recovery. Then correlate transaction-log size with flush time and recovery duration. If the log grows because the cluster cannot keep up with underlying I/O, a translog setting may only hide the real bottleneck.
Bulk Ingest, RAG, and the Throughput Trade-off
OpenSearch documented about a 60 percent throughput improvement from a bundle of indexing changes on Intel EC2 r7iz.2xlarge instances using the StackOverflow workload. The bundle included heap, refresh, merge, shard, codec, bulk, and flush-threshold changes, so the gain cannot be attributed to the transaction log alone (OpenSearch Project, 2024).
A RAG service may ingest in bursts and then serve latency-sensitive queries. Fewer flushes can help the ingest phase, while a larger log can delay shard readiness after failure. The trade-off is workload-phase dependent.
For batch imports where source data is safely retained elsewhere, teams can often accept more aggressive performance experiments. For primary systems of record, request durability and conservative thresholds usually dominate. For mixed workloads, consider operational scheduling: ingest heavily during a controlled window, restore normal refresh and durability behavior afterward, and test failure recovery before declaring the benchmark a success.
Remote Storage Is Rewriting the Durability Boundary
The bigger architectural change is remote durability. OpenSearch remote-backed storage persists index transactions to a remote store and can recover through the last acknowledged write with request durability. Its launch team described that mode as a zero-RPO design (Kale et al., 2023; OpenSearch Project, n.d.-e).
Elastic has taken a distinct Serverless path. A 2024 engineering post described uploads every 200 milliseconds or 16 MiB, with shard writes coalesced to control object-store cost. Later work added per-node buffering to reduce write amplification (Elastic, 2024; Elastic, 2025). These are architecture examples, not self-managed tuning defaults.
The practical implication is that durability tuning increasingly includes storage topology, remote upload latency, API cost, and replication design. OpenSearch node stats now expose remote-translog upload metrics, while remote-store configuration can restrict async durability. The operation log is becoming a bridge not only between memory and Lucene commits, but also between local execution and durable cloud storage.
The Future of Translogs in 2027
By 2027, transaction-log tuning is likely to become less about one universal flush threshold and more about policy choices tied to workload type and storage architecture. Elastic is already reducing local-state assumptions in Serverless, while OpenSearch remote-backed storage separates primary indexing from remote durability and segment distribution. The common direction is durable operation logging with more of the persistence burden shifted to shared object storage.
Sequence-number changes point to another trend. Elastic reported in June 2026 that time-series data stream work could reduce storage by 41 percent by trimming sequence numbers after replication for workloads that do not need their full concurrency semantics (Fernandez Castano, Leroux, & Woodward, 2026). That does not eliminate the transaction log, but it shows that recovery metadata and write-path guarantees are becoming more workload-aware.
The uncertain part is how quickly these ideas will converge across self-managed, managed, and serverless products. Operators should expect more remote-store metrics, workload-specific settings, and guardrails around risky durability modes. They should not expect the basic trade-off to disappear. Faster acknowledgement, fewer commits, stronger durability, lower storage cost, and shorter recovery time still compete with one another. The platform can automate more of that balance, but it cannot decide the business cost of losing or delaying data.
Takeaways
- Treat the transaction log as a durability mechanism first and a performance lever second.
- Do not confuse refresh, fsync, and flush; they solve different visibility and durability problems.
- Keep request durability for production data unless the loss window under async is explicitly acceptable.
- Measure translog size, uncommitted operations, flush time, recovery time, indexing throughput, and tail latency before tuning.
- Do not copy OpenSearch and Elasticsearch thresholds across products or versions without checking current documentation.
- Expect remote and object storage to become a larger part of write durability, especially in serverless and remote-backed architectures.
Conclusion
The transaction log is easy to describe and easy to tune badly. Its job is to protect acknowledged operations between Lucene commits, but every attempt to reduce write-path cost changes some other part of the system: fsync frequency, failure loss window, flush work, recovery time, disk pressure, or remote-storage traffic.
The safest operating model is conservative and measurable. Keep strong durability for irreplaceable data, let automatic flush logic do its job, and change only one variable after capturing a representative baseline. When a performance guide shows a large throughput gain, verify whether the result came from a bundle of changes rather than the transaction log alone.
Search infrastructure is also becoming more distributed. Our 2026 AI search trends report covers the retrieval systems sitting above these indexes, while the storage layer below them is moving toward remote persistence and workload-specific durability. The mechanism remains familiar. The context around it is changing quickly.
Frequently Asked Questions
What is a translog in Elasticsearch?
A translog is a per-shard transaction log that records index and delete operations after Lucene has processed them but before Elasticsearch acknowledges the request. If a shard fails before those operations are part of a Lucene commit, recovery can replay the logged operations. It functions like a write-ahead log adapted to Elasticsearch and Lucene.
Is the transaction log the same as an Elasticsearch refresh?
No. A refresh makes recent Lucene segments visible to search, but it does not perform the same durability job as a flush. The transaction log protects operations between durable Lucene commits. A flush commits Lucene state and starts a new log generation. Confusing refresh with durability is a common tuning mistake.
Should I change index.translog.durability to async?
Only when the data-loss window is acceptable and source events can be replayed or rebuilt. Async durability can reduce synchronous fsync pressure, but acknowledged writes since the last sync may be lost after a crash. For irreplaceable production data, request durability is usually the safer baseline.
What does index.translog.sync_interval control?
It controls how often the transaction log is fsynced and committed when asynchronous syncing is relevant. Elasticsearch and current OpenSearch documentation both list a 5-second default and a 100 ms minimum. A longer interval can reduce sync overhead but increases the potential loss window under async durability.
Does a larger flush threshold always improve indexing speed?
No. A larger threshold can reduce flush frequency and may improve a write-heavy benchmark, but the effect depends on storage, merge pressure, shard count, refresh settings, bulk size, and the workload. It also leaves more work to replay after failure, so recovery objectives must be tested alongside throughput.
How do I troubleshoot slow translog recovery in OpenSearch?
Start with index stats for translog, flush, recovery, indexing, merge, and store metrics. Look for rising uncommitted bytes, long flush times, storage latency, repeated shard movement, or disk pressure. Avoid assuming the log itself is the root cause. In remote-backed clusters, also inspect remote-translog upload and restore behavior.
Methodology
This analysis was built from current Elasticsearch and OpenSearch documentation, recent engineering posts, and platform-specific performance guidance. We used primary sources for translog behavior, current settings, flush semantics, index statistics, remote storage, and serverless architecture. Named practitioner context came from Elastic engineer Philipp Kahr’s 2025 benchmarking guide and the OpenSearch remote-backed-storage engineering team.
Known limitations: current OpenSearch documentation does not expose every historical transaction-log setting in the same way as older versioned tuning guides. Where a value came from OpenSearch 2.12 rather than the latest documentation set, the article labels it as version-specific. Serverless implementation details from Elastic engineering posts are also presented as architectural examples, not universal self-managed defaults.
Balanced perspective: increasing a threshold or relaxing durability can improve a benchmark under some workloads, but those gains can shift cost into recovery, failure risk, merge pressure, or remote-storage traffic. No single setting is recommended as globally optimal.
This article was drafted with AI assistance and reviewed by the Perplexity AI Editorial Team. All data, citations, and claims have been independently verified against primary sources.
References
Elastic. (n.d.-a). Translog settings. Elasticsearch Reference. Retrieved August 19, 2026. Source
Elastic. (n.d.-b). Flush data streams or indices. Elasticsearch API documentation. Retrieved August 19, 2026. Source
Elastic. (n.d.-c). Get index statistics. Elasticsearch API documentation. Retrieved August 19, 2026. Source
Elastic. (2024, September 6). Stateless: Data safety in a stateless world. Elasticsearch Labs. Source
Elastic. (2025). Elastic’s journey to build Elastic Cloud Serverless. Elastic Blog. Source
Fernandez Castano, F., Leroux, T., & Woodward, A. (2026, June 11). How Elasticsearch cut metrics storage by 41% by dropping sequence numbers after replication. Elasticsearch Labs. Source
Kahr, P. (2025, May 9). How to benchmark Elasticsearch performance with ingest pipelines and your own logs. Elastic Blog. Source
Kale, S., Singh, A., Bhargava, R., Khan, B., Ramachandra, R., & Bower, N. (2023, September 27). Improved durability in OpenSearch with remote-backed storage. OpenSearch. Source
OpenSearch Project. (n.d.-a). OpenSearch concepts: Update lifecycle. Retrieved August 19, 2026. Source
OpenSearch Project. (n.d.-b). Index settings. Retrieved August 19, 2026. Source
OpenSearch Project. (n.d.-c). Index Stats API. Retrieved August 19, 2026. Source
OpenSearch Project. (n.d.-d). Flush API. Retrieved August 19, 2026. Source
OpenSearch Project. (n.d.-e). Remote-backed storage. Retrieved August 19, 2026. Source
OpenSearch Project. (2024). Tuning your cluster for indexing speed (OpenSearch 2.12 documentation). Source
