Make your data searchable

@searchable

The @searchable directive handles streaming the data of an @model object type to the Amazon OpenSearch Service and configures search resolvers that search that information.

Migration warning: You might observe duplicate records on search operations, if you deployed your GraphQL schema using CLI version older than 4.14.1 and have thereafter updated your schema & deployed the changes with a CLI version between 4.14.1 - 4.16.1. Please use this Python script to remove the duplicate records from your OpenSearch cluster. This script indexes data from your DynamoDB Table to your OpenSearch Cluster. View an example of how to call the script with the following parameters here.

Billing warning: @searchable incurs an additional cost depending on instance size. For more information refer to the Amazon OpenSearch service pricing.

Definition

1# Streams data from DynamoDB to OpenSearch and exposes search capabilities.
2directive @searchable(queries: SearchableQueryMap) on OBJECT
3input SearchableQueryMap {
4  search: String
5}

Usage

Given the following schema an index is created for Post, if there are more types with @searchable the directive will create an index for it, and those posts in Amazon DynamoDB are automatically streamed to the post index in Amazon OpenSearch via AWS Lambda and connect a searchQueryField resolver.

1type Post @model @searchable {
2  id: ID!
3  title: String!
4  createdAt: String!
5  updatedAt: String!
6  upvotes: Int
7}

You may then create objects in DynamoDB that will be automatically streamed to lambda using the normal createPost mutation.

1mutation CreatePost {
2  createPost(input: { title: "Stream me to OpenSearch!" }) {
3    id
4    title
5    createdAt
6    updatedAt
7    upvotes
8  }
9}

And then search for posts using a match query:

1query SearchPosts {
2  searchPosts(filter: { title: { match: "Stream" } }) {
3    items {
4      id
5      title
6    }
7  }
8}

There are multiple SearchableTypes generated in the schema, based on the datatype of the fields you specify in the Post type.

The filter parameter in the search query has a searchable type field that corresponds to the field listed in the Post type. For example, the title field of the filter object, has the following properties (containing the operators that are applicable to the string type):

• eq - which uses the OpenSearch keyword type to match for the exact term.
• ne - this is the inverse operation of eq.
• matchPhrase - searches using OpenSearch's Match Phrase Query to filter the documents in the search query.
• matchPhrasePrefix - This uses OpenSearch's Match Phrase Prefix Query to filter the documents in the search query.
• multiMatch - Corresponds to the OpenSearch Multi Match Query.
• exists - Corresponds to the OpenSearch Exists Query.
• wildcard - Corresponds to the OpenSearch Wildcard Query.
• regexp - Corresponds to the OpenSearch Regexp Query.

The sort parameter can be used to specify the order of the search results, can be ascending (asc) or descending (desc), if not specified ascending order is used.

The limit parameter controls the number of search results returned. If not specified the default value is 100.

For example, you can filter using the wildcard expression to search for posts using the following wildcard query:

1query SearchPosts {
2  searchPost(filter: { title: { wildcard: "S*OpenSearch!" } }) {
3    items {
4      id
5      title
6    }
7  }
8}

The above query returns all documents whose title begins with S and ends with OpenSearch!.

Moreover you can use the filter parameter to pass a nested and/or/not condition. By default, every operation in the filter properties is AND ed. You can use the or or not properties in the filter parameter of the search query to override this behavior. Each of these operators (and, or, not properties in the filter object) accepts an array of searchable types which are in turn joined by the corresponding operator. For example, consider the following search query:

1query SearchPosts {
2  searchPost(
3    filter: {
4      title: { wildcard: "S*" }
5      or: [
6        { createdAt: { eq: "08/20/2018" } }
7        { updatedAt: { eq: "08/20/2018" } }
8      ]
9    }
10  ) {
11    items {
12      id
13      title
14    }
15  }
16}

Assuming you used the createPost mutation to create new posts with title, createdAt and updatedAt values, the above search query will return you a list of all Posts, whose title starts with S and have createdAt or updatedAt value as 08/20/2018.

Here is a complete list of searchable operations per GraphQL type supported as of today:

Known limitations

• @searchable is not compatible with DataStore but you can use it with the API category.
• @searchable is not compatible with Amazon ElasticSearch t2.micro instance as it only works with ElasticSearch version 1.5 and 2.3 and Amplify CLI only supports instances with ElasticSearch version >= 6.x.
• @searchable is not compatible with the @connection directive
• Support for adding the @searchable directive does not yet provide automatic indexing for any existing data to OpenSearch. View the feature request here.
• t2.small OpenSearch instance type is not recommended to be used in a production environment.

Configure environment OpenSearch instance type

By default Amplify CLI will configure a t2.small instance type. This is great for getting started and prototyping BUT not recommended to be used in the production environment per the OpenSearch best practices. You can configure the OpenSearch instance type per environment as follows:

1. Run amplify env add to create a new environment (e.g. "prod")
2. Edit the amplify/team-provider-info.json file and set ElasticSearchInstanceType to the instance type that works for your application

1{
2  "dev": {
3    "categories": {
4      "api": {
5        "<your-api-name>": {
6          "ElasticSearchInstanceType": "t2.small.elasticsearch"
7        }
8      }
9    }
10  },
11  "prod": {
12    "categories": {
13      "api": {
14        "<your-api-name>": {
15          "ElasticSearchInstanceType": "t2.medium.elasticsearch"
16        }
17      }
18    }
19  }
20}

3. Deploy your changes with amplify push

Learn more about Amazon OpenSearch Service instance types here.

Backfill your OpenSearch index from your DynamoDB table

The following Python script creates an event stream of your DynamoDB records and sends them to your OpenSearch Index. This will help you backfill your data should you choose to add @searchable to your @model types at a later time.

Example of calling the script:

1python3 ddb_to_es.py \
2  --rn 'us-west-2' \ # Use the region in which your table and OpenSearch domain reside
3  --tn 'Post-XXXX-dev' \ # Table name
4  --lf 'arn:aws:lambda:us-west-2:123456789xxx:function:DdbToEsFn-<api__id>-dev' \ # Lambda function ARN, find the DdbToEsFn in your Lambda functions list, copy entire ARN
5  --esarn 'arn:aws:dynamodb:us-west-2:123456789xxx:table/Post-<api__id>-dev/stream/2019-20-03T00:00:00.350' # Event source ARN, copy the full DynamoDB table ARN