Link Search Menu Expand Document Documentation Menu
Documentation

Using raw vectors for neural sparse search

If you’re using self-hosted sparse embedding models, you can ingest raw sparse vectors and use neural sparse search.

Tutorial

This tutorial consists of the following steps:

  1. Ingest sparse vectors
    1. Create an index
    2. Ingest documents into the index
  2. Search the data using raw sparse vector.

Step 1: Ingest sparse vectors

Once you have generated sparse vector embeddings, you can directly ingest them into OpenSearch.

Step 1(a): Create an index

In order to ingest documents containing raw sparse vectors, create a rank features index:

PUT /my-nlp-index
{
  "mappings": {
    "properties": {
      "id": {
        "type": "text"
      },
      "passage_embedding": {
        "type": "rank_features"
      },
      "passage_text": {
        "type": "text"
      }
    }
  }
}

Step 1(b): Ingest documents into the index

To ingest documents into the index created in the previous step, send the following request:

PUT /my-nlp-index/_doc/1
{
  "passage_text": "Hello world",
  "id": "s1",
  "passage_embedding": {
    "hi" : 4.338913,
    "planets" : 2.7755864,
    "planet" : 5.0969057,
    "mars" : 1.7405145,
    "earth" : 2.6087382,
    "hello" : 3.3210192
  }
}

Step 2: Search the data using a sparse vector

To search the documents using a sparse vector, provide the sparse embeddings in the neural_sparse query:

GET my-nlp-index/_search
{
  "query": {
    "neural_sparse": {
      "passage_embedding": {
        "query_tokens": {
          "hi" : 4.338913,
          "planets" : 2.7755864,
          "planet" : 5.0969057,
          "mars" : 1.7405145,
          "earth" : 2.6087382,
          "hello" : 3.3210192
        }
      }
    }
  }
}

To learn more about improving retrieval time for neural sparse search, see Accelerating neural sparse search.

350 characters left

Have a question? .

Want to contribute? or .