Elasticsearch.

Elasticsearch is a distributed search and analytics engine built on Apache Lucene. It stores data as JSON documents behind a RESTful API, and new data becomes searchable in near real time, within about one second. It scales horizontally by adding nodes, and it keeps copies of the data on several nodes for reliability.

Its main use cases are:

• Full-text search on websites, documents and e-commerce products.
• Log analytics on application, system and security logs, often with the ELK stack of Elasticsearch, Logstash and Kibana.
• Real-time analytics, such as business metrics and application performance monitoring.
• Data aggregation for complex analysis and reporting.
• Geospatial search and analytics by location.

You search and analyze the data with the Query DSL, a JSON-based query language.

Elasticsearch stores data in indices, splits each index into shards, and spreads the shards over the nodes of a cluster. Its building blocks are:

• A node is a single running instance of Elasticsearch.
• A cluster is a collection of nodes that work together.
• An index is a collection of documents with similar characteristics. In relational terms it is closer to a table than to a database.
• A document is the basic unit of information, stored as JSON.
• A shard is a subset of an index's data. Elasticsearch distributes the shards across the nodes.
• A replica is a copy of a primary shard. It adds redundancy and also serves searches, which raises search throughput.

A replica never sits on the same node as its primary. In the layout below, Index A has two primary shards and Index B has one, and every primary has a replica on another node. So losing any one node loses no data.

Cluster
├── Node 1
│   ├── Index A (Primary Shard 0)
│   └── Index A (Replica Shard 1)
├── Node 2
│   ├── Index A (Primary Shard 1)
│   └── Index B (Primary Shard 0)
└── Node 3
    ├── Index A (Replica Shard 0)
    └── Index B (Replica Shard 0)

Elasticsearch is built to search and analyze data across many machines, while a relational database stores records and changes them safely in transactions. Elasticsearch keeps denormalized JSON documents spread over nodes. A relational database keeps normalized rows in tables and joins them at query time.

Elasticsearch is the better fit for search, analytics and unstructured data. A relational database is the better fit for transactional work that needs ACID properties, such as orders and payments. Many systems use both. The database stays the source of truth, and Elasticsearch holds a copy of the data for search.

You perform CRUD on Elasticsearch documents with the document APIs. Each operation is one HTTP request to a path made of the index name and the document ID.

To create a document, or replace it if the ID exists, send PUT with the JSON body. This is called indexing a document.

PUT /users/_doc/1
{
  "name": "John Doe",
  "email": "john@example.com",
  "age": 30
}

To read it, send GET to the same path.

GET /users/_doc/1

To change only some fields, use the update API. It merges the partial doc into the stored document.

POST /users/_update/1
{
  "doc": {
    "age": 31
  }
}

To delete it, send DELETE to the same path.

DELETE /users/_doc/1

For many documents, the bulk API sends many operations in one request. The body is newline-delimited JSON, with an action line before each document. Bulk requests are much faster than one request per document, which matters for large loads.

POST /_bulk
{"index": {"_index": "users", "_id": "1"}}
{"name": "Alice", "age": 25}
{"index": {"_index": "users", "_id": "2"}}
{"name": "Bob", "age": 30}

The index operation writes a document whether or not its ID already exists, while the create operation writes it only if the ID is new.

• Index, PUT /index/_doc/id, creates a new document or replaces the entire existing one. Each replace increments the document's _version.
• Create, PUT /index/_create/id, only creates a new document. If a document with the same ID exists, it fails with a 409 Conflict error. The index API gives the same check with op_type=create.

Use create when a duplicate ID would be a bug, such as writing events that must be stored once. Use index when the latest version should win.

# Index - will replace if exists
PUT /users/_doc/1
{"name": "John", "age": 30}

# Create - will fail if ID 1 exists
PUT /users/_create/1
{"name": "Jane", "age": 25}

Run in this order, the second request fails, because the first one already wrote ID 1.