Technology Sharing

[Elasticsearch] The evolution and selection of open source search technology: Elasticsearch and OpenSearch

2024-07-12

한어Русский языкEnglishFrançaisIndonesianSanskrit日本語DeutschPortuguêsΕλληνικάespañolItalianoSuomalainenLatina

1. Historical Development

In 2010, Apache 2.0 2.0 2.0 After the open source agreement is released,Elasticsearch Quickly becoming the most popular company in the worldsearch engine

Elasticsearch Often withLogstash andKibana Deployed together, this combination is calledElasitc Stack, used to enable log analytics use cases including application observability, security log analysis, and understanding user behavior.

In 2015, Amazon used this open source protocol to launch Amazon Elasticsearch ServiceAmazon ES), a cloud hosting service that allows AWS customers to launch scalable Elasticsearch Cluster, connect data sources to cluster endpoints, and load, process, analyze, or visualize data in the cloud.

However, Elastic eventually objected to Amazon's use of its products and trademarks, filing a lawsuit against the tech giant in 2019 for alleged trademark infringement and false advertising.

insert image description here

🚀《Elastic and Amazon reach agreement on trademark infringement lawsuit

Elastic and Amazon settled this litigation in 2021, resulting in two important developments:

  • In January 2021, Elastic announced that 7.11 Starting with version 1, Elasticsearch will be based onServer Side Public LicenseSSPLThis change prevented Amazon and other companies from offering Elasticsearch services without working directly with Elastic.
  • In April 2021, Amazon announced that it would move away from the last open source version of Elasticsearch (7.10.2) branch, launching a new open source search engine project. Elasticsearch The name can no longer be used, the new project is namedOpenSearchAmazon also offers OpenSearch as a cloud service called Amazon OpenSearch Service

🚀 OpenSearch 官网:https://opensearch.org/

Here is a reminder that Alibaba Cloud's early search engine was also called OpenSearch. Coincidentally, Alibaba Cloud's open search service is also called OpenSearch. Its address can be found at:https://developer.aliyun.com/article/67156

Alibaba Cloud OpenSearchOpen search service) is a set of self-service, customizable cloud search services launched by Alibaba Cloud. The original intention is to platformize and service the search engine technology accumulated by Alibaba for nearly 10 years, and open it to a large number of developers, lowering the threshold for realizing professional search products, and allowing developers to easily have professional search products similar to the search tools of Taobao, Tmall and other applications at a lower cost.

This article introduces OpenSearch, which is a branch of Elasticserach, not Alibaba Cloud's OpenSearch. Please note this.

In the following sections, we’ll take a closer look at how these two search solutions have evolved over the past nearly 3 years and which one is better suited for your AWS log analysis or enterprise search application.

The following similarities and differences are mainly from: Elastic official website, OpenSearch official website and related blogs.

2. Similarities between OpenSearch and Elasticsearch

OpenSearch is developed from a relatively advanced fork of Elasticsearch, so the two applications share all the basic features of search, analysis, and dashboards.

insert image description here
Both OpenSearch and Elasticsearch provideMulti-tenant architecture andresearch allDistributed Search Capabilities analysis engine.

Therefore, both can be used to satisfy Site SearchEnterprise Search andLog analysis Use case for .

3. Differences between OpenSearch and Elasticsearch

When comparing OpenSearch and Elasticsearch, there are several key differences:

  • Version
  • license
  • Community
  • Function
  • safety
  • performance
  • price

3.1 Versions are very different

The following diagram shows the independent development paths and release cadences of the two search engine solutions, Elasticsearch and OpenSearch.

insert image description here

  • Elasticsearch 1.x2.x5.x6.x7.x8.x Version updates, eventually developing to the expected release in 20249.x Version.
  • OpenSearch from 1.0 The version will be released starting in July 2021 and will be gradually updated to2.x series, of which2.6 The version is expected to be the last in 20242.x Series version, planned to be released later3.x series.

3.2 Different licenses

Before 2021, Elasticsearch was available on Apache 2.0 2.0 2.0 License. This license allows users to use, distribute or modify the software for any purpose, and to distribute modified versions of Elasticsearch without paying any royalties.

After dispute with Amazon, Elasticsearch now runs on SSPL andElastic License After this change, Elasticsearch is no longer considered strictly open source, as the license now requires anyone providing Elasticsearch functionality to third parties to release the entire source code and all the APIs that third parties need to run it.

OpenSearch currently uses Apache 2.0 License Authorization.

🚀 简述一下 SSPL 协议和常规开源的不同:SSPL 协议要求如果你提供基于此协议软件的服务,必须公开整个服务的源代码,而常规开源软件不需要。

At this point, you may be thinking, we are using Elasticsearch and the source code is not open to the public!

It is like this. In China, there are only a handful of companies that can truly redevelop and commercialize code based on open source code, except for a few large companies such as BAT and ByteDance.

In layman's terms, this is an agreement targeting the commercialization practices of large companies like Amazon!

Some open source software using the SSPL protocol are as follows:

name of softwareUse Agreement
time
MongoDBSSPLSSPL was launched in October 2018
GraylogSSPLIn November 2020, it was announced that versions 4.0 and later will use the SSPL.
Elasticsearch & KibanaSSPL & Elastic LicenseIn January 2021, Elastic NV announced that future versions will be dual-licensed under the SSPL and Elastic License

3.3 Different communities

Both OpenSearch and Elasticsearch have healthy, active, and loyal communities, so it's unlikely that either search engine will fall too far behind in feature delivery or become unsupported any time soon.

insert image description here

insert image description here
I took a look and found that the number of questions and replies in the OpenSearch forum has not yet reached a certain scale.

3.4 Different functions

Amazon initially forked Elasticsearch 7.10.2 When creating OpenSearch, you must first remove all incompatible Apache 2.0 2.0 2.0 The license code.

insert image description here

This includes disabling all paid features and removing the entire Elastic X-Pack code.

FunctionElasticsearchOpenSearch
Remark
licenseSSPL / Elastic LicenseApache License 2.0OpenSearch maintains a fully open source license
security functionSome of these features are paidMost open source availableOpenSearch leans toward open-sourcing its security features
Alerting and anomaly detectionPaid and open source versionsOpen source availableOpenSearch provides open source alerting and anomaly detection
SQL supporthavehaveOpenSearch inherits the SQL functionality of Open Distro
Machine Learning SupportPaid FeaturesAvailable as an open source pluginOpenSearch community provides ML plugins
Client CompatibilityFully support our productsCompatible with Elasticsearch 7.10OpenSearch remains compatible with old Elasticsearch clients
Community SupportLed by ElasticCommunity DrivenOpenSearch encourages community contributions and feedback
Plugin EcosystemA relatively closed ecosystemOpen plugin ecosystemOpenSearch promotes open source development of plugins
Tracking AnalysisPaid FeaturesOpen source availableOpenSearch provides open source tracking and analysis capabilities
Long-term support versionsupplysupplyOpenSearch promises to provide long-term supported versions

Elastic GitHub:https://github.com/elastic

insert image description here

OpenSearch GitHub:https://github.com/opensearch-project

insert image description here

3.5 Different security

Both Elasticsearch and OpenSearch come with many of the same security features.

However, these features are included in OpenSearch, while some features are only available to paying customers in Elasticsearch.

As OpenSearch official website introduces: it is secure by default (system default).

insert image description here

Elastic paid features list:https://www.elastic.co/cn/pricing

insert image description here

3.6 Different performance

If you are running large-scale search operations, you might be interested in which of these two search engines provides faster queries.

Elastic recently published a blog titled “Elasticsearch vs. OpenSearch: Uncovering the performance gap”.

🚀 文章地址:https://www.elastic.co/cn/blog/elasticsearch-opensearch-performance-gap

Elastic's official website document emphasizes: Since an early version of Elasticsearch came out, OpenSearch has not kept up with the pace of innovation. Since OpenSearch GA, Elasticsearch has surpassed 4 4 4 times.

Elastic's official comparison emphasizes, as shown in the figure below - Elasticsearch has advantages in various key performance indicators and saves more storage resources.

insert image description here

  • Text queryText Querying): Elasticsearch is faster than OpenSearch in text query performance 76 % 76% 76%
  • SortingSorting): Elasticsearch is faster than OpenSearch in sorting operations 58 % 58% 58%
  • Date HistogramDate Histogram): Elasticsearch appears to be faster than OpenSearch when performing date histogram aggregation 81 % 81% 81% performance.
  • TermsTerms): Elasticsearch performs faster than OpenSearch when processing terms queries 61 % 61% 61%
  • scopeRanges): Elasticsearch is faster than OpenSearch in range queries 42 % 42% 42% performance.
  • Elasticsearch uses storage Less than OpenSearch 37 % 37% 37%

This means that when Elasticsearch handles the same task, not only the query speed is faster, but also the storage efficiency is better.

See the blog for the following screenshots:https://www.elastic.co/cn/blog/migrating-1-billion-log-lines-opensearch-elasticsearch

insert image description here
The left side is Opensearch, and the right side is Elasticsearch. It is obvious that Elasticsearch saves 23 % 23% 23% storage space.

3.7 Different prices

Self-managed versions of Elasticsearch and OpenSearch are available for free download, but users are required to use their own hardware and computing resources to install, manage and operate them.

Elasticsearch is mostly used by small and medium-sized enterprises in China, and the ordinary free basic functions are basically sufficient.

There are also many large companies such as some commercial banks that use the commercial functions of Elasticsearch.

Another new feature in China is that cloud vendors have worked closely with Elastic to launch Elasticsearch versions. For example, Alibaba Cloud Elasticsearch and Tencent Cloud Elasticsearch are also widely used.
insert image description here

insert image description here

3.8 Both can import each other

OpenSearch migration to Elasticsearch see:

Elasticsearch migration to OpenSearch see:

4. Summary

OpenSearch has been released for more than 3 years. Elasticsearch and OpenSearch have many differences, but they are ultimately derived from the same version: Elasticsearch 7.10.2

We do not provide direct recommendations between Elasticsearch and Amazon OpenSearch, as the most appropriate choice will depend on the unique needs and environment of your project or product. However, the data in this article will provide you with a reference.

Readers are advised to make decisions based on their own specific circumstances and preferences, and consider functional requirements, cost-effectiveness, long-term technical support, or their own ability to optimize architecture and performance.