liveBook · Manning

Bulk API

As explained in Elasticsearch in Action, Second Edition

The Elasticsearch _bulk API accepts multiple document indexing, update, or deletion actions in a single request.

The Elasticsearch _bulk API sends many document actions in one POST request. Use it when importing a catalog, loading records from a database, or making many document changes at once; sending documents one by one becomes cumbersome for larger data sets. A bulk request can combine index, create, update, and delete actions, reducing repeated trips to the server. section 5.0 section 5.6

Bulk requests use newline-delimited JSON (NDJSON). An action-and-metadata line comes first; for index, create, and update, the next line supplies the document source or update payload. Metadata commonly identifies the target index and document ID. Each line, including the last one, must end with a newline: the endpoint is newline-sensitive. Delete is different: its action line needs no following source line. section 5.6.1 section 5.6.3

Index several movie documents

Send the index name in the URL to avoid repeating it in every action line. Empty index metadata lets Elasticsearch generate document IDs. POST movies/_bulk {"index":{}} {"title":"Mission Impossible","release_date":"1996-07-05"} {"index":{}} {"title":"Mission Impossible II","release_date":"2000-05-24"} Each action/source pair is one document operation. Use explicit _id metadata when a known document ID matters; otherwise, omit it for a system-generated ID. section 5.6.2

Listing 5.45 Bulk request with mixed bag of requests
POST _bulk
{"index":{"_index":"books"}}                           #1
{"title": "Elasticsearch in Action"}
{"create":{"_index":"flights", "_id":"101"}}           #2
{"title": "London to Bucharest"}
{"index":{"_index":"pets"}}                            #3
{"name": "Milly","age_months": 18}
{"delete":{"_index":"movies", "_id":"101"}}            #4
{ "update" : {"_index":"movies", "_id":"1"} }          #5
{ "doc" : {"title" : "The Godfather (Original)"} }

A single request can target different indices and mix index, create, delete, and update actions. For updates, place changed fields inside a doc object; treat delete carefully because it can remove data permanently. section 5.6.3

Bulk indexing is a production-oriented tool, not a fixed-size recipe. Test batch sizes against the cluster to find an appropriate size. For large overnight ingestion, increasing the refresh setting during indexing can improve disk-usage performance; assess that adjustment in the target environment. section 15.2.2

Bulk operations write into an existing index unless the request targets another one. Loading richer documents can add fields to the indexed object structure, so use a separate index when existing data must remain untouched. section 2.3.3

How Elasticsearch in Action, Second Edition covers it

How deep it goes
Explains NDJSON request structure, metadata, indexing variants, mixed actions, and batch-performance tuning.
Examples
Demonstrates books and movie imports, Tom Cruise movie batches, mixed entities, an update to Rush Hour, and a movie-review deletion.
What it assumes
Builds from individual document APIs and introduces bulk request formatting before batch operations.
What you can do after
Apply it via the Elasticsearch _bulk endpoint to batch index, create, update, and delete documents using NDJSON.
How it connects
Connects document indexing, document IDs, NDJSON, create, update, delete, refresh settings, and reindexing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Elasticsearch in Action, Second Edition. The section links point to the parts of the book it draws on.