Vector search
OpenSearch is a comprehensive search platform that supports a variety of data types, including vectors. OpenSearch vector database functionality is seamlessly integrated with its generic database function.
In OpenSearch, you can generate vector embeddings, store those embeddings in an index, and use them for vector search. Choose one of the following options:
- Generate embeddings using a library of your choice before ingesting them into OpenSearch. Once you ingest vectors into an index, you can perform a vector similarity search on the vector space. For more information, see Working with embeddings generated outside of OpenSearch.
- Automatically generate embeddings within OpenSearch. To use embeddings for semantic search, the ingested text (the corpus) and the query need to be embedded using the same model. Neural search packages this functionality, eliminating the need to manage the internal details. For more information, see Generating vector embeddings within OpenSearch.
Working with embeddings generated outside of OpenSearch
After you generate vector embeddings, upload them to an OpenSearch index and search the index using vector search. For a complete example, see Example.
k-NN index
To build a vector database and use vector search, you must specify your index as a k-NN index when creating it by setting index.knn to true:
PUT test-index
{
"settings": {
"index": {
"knn": true,
"knn.algo_param.ef_search": 100
}
},
"mappings": {
"properties": {
"my_vector1": {
"type": "knn_vector",
"dimension": 1024,
"method": {
"name": "hnsw",
"space_type": "l2",
"engine": "nmslib",
"parameters": {
"ef_construction": 128,
"m": 24
}
}
}
}
}
}
k-NN vector
You must designate the field that will store vectors as a knn_vector field type. OpenSearch supports vectors of up to 16,000 dimensions, each of which is represented as a 32-bit or 16-bit float.
To save storage space, you can use byte or binary vectors. For more information, see Lucene byte vector and Binary k-NN vectors.
k-NN vector search
Vector search finds the vectors in your database that are most similar to the query vector. OpenSearch supports the following search methods:
-
Approximate search (approximate k-NN, or ANN): Returns approximate nearest neighbors to the query vector. Usually, approximate search algorithms sacrifice indexing speed and search accuracy in exchange for performance benefits such as lower latency, smaller memory footprints, and more scalable search. For most use cases, approximate search is the best option.
-
Exact search (exact k-NN): A brute-force, exact k-NN search of vector fields. OpenSearch supports the following types of exact search:
- Exact k-NN with scoring script: Using the k-NN scoring script, you can apply a filter to an index before executing the nearest neighbor search.
- Painless extensions: Adds the distance functions as Painless extensions that you can use in more complex combinations. You can use this method to perform a brute-force, exact k-NN search of an index, which also supports pre-filtering.
Approximate search
OpenSearch supports several algorithms for approximate vector search, each with its own advantages. For complete documentation, see Approximate search. For more information about the search methods and engines, see Method definitions. For method recommendations, see Choosing the right method.
To use approximate vector search, specify one of the following search methods (algorithms) in the method parameter:
- Hierarchical Navigable Small World (HNSW)
- Inverted File System (IVF)
Additionally, specify the engine (library) that implements this method in the engine parameter:
The following table lists the combinations of search methods and libraries supported by the k-NN engine for approximate vector search.
| Method | Engine |
|---|---|
| HNSW | NMSLIB, Faiss, Lucene |
| IVF | Faiss |
Engine recommendations
In general, select NMSLIB or Faiss for large-scale use cases. Lucene is a good option for smaller deployments and offers benefits like smart filtering, where the optimal filtering strategy—pre-filtering, post-filtering, or exact k-NN—is automatically applied depending on the situation. The following table summarizes the differences between each option.
| NMSLIB/HNSW | Faiss/HNSW | Faiss/IVF | Lucene/HNSW | |
|---|---|---|---|---|
| Max dimensions | 16,000 | 16,000 | 16,000 | 16,000 |
| Filter | Post-filter | Post-filter | Post-filter | Filter during search |
| Training required | No | No | Yes | No |
| Similarity metrics | l2, innerproduct, cosinesimil, l1, linf | l2, innerproduct | l2, innerproduct | l2, cosinesimil |
| Number of vectors | Tens of billions | Tens of billions | Tens of billions | Less than 10 million |
| Indexing latency | Low | Low | Lowest | Low |
| Query latency and quality | Low latency and high quality | Low latency and high quality | Low latency and low quality | High latency and high quality |
| Vector compression | Flat | Flat Product quantization | Flat Product quantization | Flat |
| Memory consumption | High | High Low with PQ | Medium Low with PQ | High |
Example
In this example, you’ll create a k-NN index, add data to the index, and search the data.
Step 1: Create a k-NN index
First, create an index that will store sample hotel data. Set index.knn to true and specify the location field as a knn_vector:
PUT /hotels-index
{
"settings": {
"index": {
"knn": true,
"knn.algo_param.ef_search": 100,
"number_of_shards": 1,
"number_of_replicas": 0
}
},
"mappings": {
"properties": {
"location": {
"type": "knn_vector",
"dimension": 2,
"method": {
"name": "hnsw",
"space_type": "l2",
"engine": "lucene",
"parameters": {
"ef_construction": 100,
"m": 16
}
}
}
}
}
}
Step 2: Add data to your index
Next, add data to your index. Each document represents a hotel. The location field in each document contains a vector specifying the hotel’s location:
POST /_bulk
{ "index": { "_index": "hotels-index", "_id": "1" } }
{ "location": [5.2, 4.4] }
{ "index": { "_index": "hotels-index", "_id": "2" } }
{ "location": [5.2, 3.9] }
{ "index": { "_index": "hotels-index", "_id": "3" } }
{ "location": [4.9, 3.4] }
{ "index": { "_index": "hotels-index", "_id": "4" } }
{ "location": [4.2, 4.6] }
{ "index": { "_index": "hotels-index", "_id": "5" } }
{ "location": [3.3, 4.5] }
Step 3: Search your data
Now search for hotels closest to the pin location [5, 4]. This location is labeled Pin in the following image. Each hotel is labeled with its document number.
To search for the top three closest hotels, set k to 3:
POST /hotels-index/_search
{
"size": 3,
"query": {
"knn": {
"location": {
"vector": [
5,
4
],
"k": 3
}
}
}
}
The response contains the hotels closest to the specified pin location:
{
"took": 1093,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"skipped": 0,
"failed": 0
},
"hits": {
"total": {
"value": 3,
"relation": "eq"
},
"max_score": 0.952381,
"hits": [
{
"_index": "hotels-index",
"_id": "2",
"_score": 0.952381,
"_source": {
"location": [
5.2,
3.9
]
}
},
{
"_index": "hotels-index",
"_id": "1",
"_score": 0.8333333,
"_source": {
"location": [
5.2,
4.4
]
}
},
{
"_index": "hotels-index",
"_id": "3",
"_score": 0.72992706,
"_source": {
"location": [
4.9,
3.4
]
}
}
]
}
}
Vector search with filtering
For information about vector search with filtering, see k-NN search with filters.
Generating vector embeddings in OpenSearch
Neural search encapsulates the infrastructure needed to perform semantic vector searches. After you integrate an inference (embedding) service, neural search functions like lexical search, accepting a textual query and returning relevant documents.
When you index your data, neural search transforms text into vector embeddings and indexes both the text and its vector embeddings in a vector index. When you use a neural query during search, neural search converts the query text into vector embeddings and uses vector search to return the results.
Choosing a model
The first step in setting up neural search is choosing a model. You can upload a model to your OpenSearch cluster, use one of the pretrained models provided by OpenSearch, or connect to an externally hosted model. For more information, see Integrating ML models.
Neural search tutorial
For a step-by-step tutorial, see Neural search tutorial.
Search methods
Choose one of the following search methods to use your model for neural search:
-
Semantic search: Uses dense retrieval based on text embedding models to search text data.
-
Hybrid search: Combines lexical and neural search to improve search relevance.
-
Multimodal search: Uses neural search with multimodal embedding models to search text and image data.
-
Neural sparse search: Uses neural search with sparse retrieval based on sparse embedding models to search text data.
-
Conversational search: With conversational search, you can ask questions in natural language, receive a text response, and ask additional clarifying questions.