How to migrate from Amazon CloudSearch to Amazon OpenSearch Serverless

If you run search on Amazon CloudSearch, now is the time to plan your migration to Amazon OpenSearch Serverless. Modern search has moved on to capabilities beyond what CloudSearch provides: semantic and hybrid search, Retrieval Augmented Generation (RAG), and agentic search. OpenSearch Serverless gives you all of these with automatic scaling on a pay-for-what-you-use basis. You don’t need to choose or maintain infrastructure. OpenSearch Serverless maintains the hands-off, operational simplicity of CloudSearch.

This post shows you how to migrate your CloudSearch domain to an Amazon OpenSearch Serverless collection. We walk you through assessing your CloudSearch configuration, creating an OpenSearch Serverless collection with explicit index mappings, converting your documents and queries, configuring security policies, loading your data with Amazon OpenSearch Ingestion, and validating the migration before cutting over.

Key differences to note

Prerequisites

To follow along with this post, you need the following:

• An AWS account.
• An existing Amazon CloudSearch domain with indexed data.
• Source data available in a durable store such as Amazon Simple Storage Service (Amazon S3) or Amazon DynamoDB (CloudSearch doesn’t provide a built-in export or backup feature, so your original source data is required to re-ingest into OpenSearch).
• AWS Identity and Access Management (IAM) permissions to create and manage Amazon OpenSearch Serverless collections, encryption policies, network policies, and data access policies.
• An Amazon OpenSearch Ingestion pipeline (or alternative ingestion method) for loading data.

Plan the migration

Planning is where you decide what success means: minimal downtime, no data loss, current functionality preserved, and custom configurations carried over. You don’t need to plan for infrastructure because OpenSearch Serverless provisions and scales compute for you. Your main planning task is to assess your current CloudSearch configuration so you can reproduce its behavior on the target.

Document your existing setup from the Amazon CloudSearch console. Record the current instance type, the partition count, and the replication count. Capture the total document count and overall data size, and record every field definition, including field types and the search, facet, and sort settings for each field. Note any analyzers, synonyms, stopwords, or custom rank expressions. Note whether you use the 2011 or the 2013 CloudSearch API version, because the 2013 API added faceting and filtering features that change how you model the target.

OpenSearch Serverless is the right target for most CloudSearch workloads, but not all of them. If your workload needs very low read-after-write latency (a short refresh interval), tight and predictable query response times, or direct control over instance configuration, choose an Amazon OpenSearch Service managed clusters deployment instead and size it from your workload profile.

The migration involves four main concerns: your source data format, your queries, your field definitions, and your access policies. Before you plan the details, it helps to see the whole migration at once. The following diagram maps the migration across four phases: your source CloudSearch environment, the migration pipeline that converts and moves your data, the OpenSearch Serverless target, and cutover and operations.

Figure 1: The migration workflow across four phases

In the source environment, you assess your CloudSearch configuration and back up your source data (Amazon S3, Amazon DynamoDB, or another store). Note the Source Data Format (SDF), the URL-based query syntax, and the IAM access policies you need to carry over. In the migration pipeline, you map field types, convert the data format from CloudSearch JSON to OpenSearch-compatible JSON, convert your queries to the OpenSearch query domain-specific language (DSL), configure security, bulk-ingest the data, and validate the result. The OpenSearch Serverless target holds the collection, index mappings, ingested documents, and the encryption, network, and data access policies, and it scales with your workload on a pay-per-use basis. In cutover and operations, you update your application to the new endpoint and clients, monitor with Amazon CloudWatch, and decommission CloudSearch once no traffic remains.

Model your data in OpenSearch Service

OpenSearch Service uses index mappings to define the fields and data types in an index. Because you know your CloudSearch schema, define the target mapping explicitly when you create the index. Create the index and set its mapping in a single request, and set dynamic to strict so OpenSearch rejects any document that contains a field you did not define. Strict mapping catches schema drift at ingest time, avoiding the default OpenSearch behavior of creating new mappings for undefined fields.

PUT /imdb_movies
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": { "type": "keyword" }
        }
      },
      "genres": { "type": "keyword" },
      "rating": { "type": "float" },
      "release_date": { "type": "date" }
      ...
    }
  }
}

Field type mapping

The following table maps CloudSearch field types to their OpenSearch Service equivalents.

Two mapping details deserve attention. First, pick the smallest numeric type that fits your data rather than copying the widths CloudSearch uses. CloudSearch stores integers as 64-bit values, but few datasets hold numbers that large. A long or a double consumes more disk than an integer, a short, or a float with no benefit when the values are small. Evaluate the actual range of each field and choose the narrowest type that holds it. Reserve long for values that genuinely exceed the roughly 2.1 billion ceiling of integer, and use float instead of double unless you need double precision. Smaller types shrink your index and speed up queries.

Second, if you sort or aggregate on a text field, add a keyword sub-field. The preceding example mapping has a keyword subfield for the title field. You access the field using dot notation: title.keyword. OpenSearch doesn’t sort or aggregate analyzed text fields by default.

As noted earlier, if you run several CloudSearch domains, model each one as a separate index within a single OpenSearch Serverless collection to consolidate them.

Move your data

Migrating to OpenSearch Service is a re-ingestion: you convert your source documents and index them into the collection you created. CloudSearch doesn’t provide a built-in backup or snapshot feature. It relies on the documents you send through the indexing process, so before you migrate, make sure your source data is available in a durable store such as Amazon S3, Amazon DynamoDB, or another database.

The conversion is a format translation. CloudSearch accepts data in SDF as JSON or XML, where a document batch is a collection of add and delete operations. The JSON that CloudSearch uses differs from the JSON that OpenSearch Service expects, so you must transform each source document into an OpenSearch document whose fields match the index mapping you defined earlier. Handle the same details the mapping calls out: emit each numeric value so it fits the narrow type you chose for its field rather than a wide long or double, format dates to match your date mapping, and drop or rename any field that your strict mapping doesn’t define.

Figure 2: CloudSearch batch format (left) compared to OpenSearch batch format (right)

You can write a small conversion script. Have the script write its output to an Amazon S3 bucket so the converted documents live in a durable store you can re-ingest from as many times as you need.

With your converted documents in Amazon S3, use Amazon OpenSearch Ingestion to load them. Amazon OpenSearch Ingestion is a feature of Amazon OpenSearch Service that you can use to ingest, filter, transform, enrich, and route data to an Amazon OpenSearch Service domain or an OpenSearch Serverless collection. Configure an OpenSearch Ingestion pipeline with an Amazon S3 source (you can use an OpenSearch Ingestion blueprint to get started) that reads your converted documents. Let its built-in processors apply any final transformation before the pipeline writes to your collection. A managed pipeline reading from Amazon S3 gives you a repeatable, restartable load without operating ingestion infrastructure, which makes it the recommended path for most migrations.

If you prefer to load data directly, OpenSearch Service exposes a REST API, so you can index documents with a standard client such as curl or with the OpenSearch client libraries for many languages. Direct indexing is convenient for a small dataset or a quick test, but an Amazon S3 source with OpenSearch Ingestion is the better choice for a production migration.

Convert your queries

CloudSearch uses a URL-based query format. You pass a query parameter in the URL and submit either a simple string search or a JSON-formatted query. OpenSearch Service uses a REST API and the OpenSearch query DSL in the request body, which gives you compound queries, function scoring, and richer relevance control. You can use generative AI coding assistants to help with this translation. Provide your CloudSearch query patterns, and the model generates the equivalent OpenSearch query DSL, which you then validate against your test cases.

Query syntax changes

CloudSearch appends parameters such as sort to the query URL, while OpenSearch expresses sorting, filtering, and boosting as explicit elements of the request body. For example, a title search for “shakespeare” in CloudSearch looks like the following.

https://my-cloudsearch-domain.us-east-1.cloudsearch.amazonaws.com/2013-01-01/search?q=shakespeare&size=10

The equivalent query in OpenSearch Service uses the query DSL.

GET /imdb_movies/_search
{
  "query": {
    "match": { "title": "shakespeare" }
  }
}

To keep result sets consistent after migration, set the default operator to AND in OpenSearch to match the default query behavior of CloudSearch. The following table shows common CloudSearch query patterns and their OpenSearch Service equivalents, using a sample IMDB movies dataset.

Sorting and boosting

Boosting is useful when you want certain fields or terms to carry more weight in relevance scoring. A higher boost value means the term contributes more to the score. OpenSearch also supports sorting by _score (relevance), which is the default when you specify no sort. For the full query language, see the OpenSearch query DSL documentation.

Configure security

CloudSearch uses AWS Identity and Access Management policies to control access to its configuration and domain service APIs. You attach user-based policies to an IAM role, user, or group, and the document, search, and suggest actions in those policies control access to the CloudSearch APIs.

OpenSearch Serverless applies security through policies at several layers.

• Collections: Encrypted at rest by default, using either an AWS owned key or a customer managed key defined in an encryption policy.
• Network policies: Define whether a collection is reachable privately through a virtual private cloud (VPC) endpoint or over the internet.
• Data access policies: Control which IAM principals and Security Assertion Markup Language (SAML) identities can create indexes and read or write data in the collection.

Amazon OpenSearch Service provisioned domains also offer fine-grained access control, with role-based access control and security at the index, document, and field level. For OpenSearch Serverless, data access policies provide collection-level and index-level permissions, controlling which IAM principals and SAML identities can create, read, or write data within a collection.

Validate the migration

Validation confirms that the migration is complete and correct before you send production traffic to OpenSearch Serverless. Work through five kinds of validation.

• Documents: Check your document count. Your OpenSearch Serverless indexes should have the same count as your CloudSearch indexes.
• Queries: Translate your most important queries and run them manually against your collection. Spot check the output for the presence of important results.
• Ranking: Check the order of results, especially for queries with custom rank functions or field weighting. Results might not match exactly, so look for anything that’s incorrect.
• Latency: Ideally you should tee your production traffic to your Serverless collection to get real latency metrics. Worst case, generate at least 100,000 synthetic queries across all your query types and run them. Monitor OpenSearch Compute Unit (OCU) consumption with Amazon CloudWatch to understand your cost profile.

To validate search functionality, run the same query against both systems and compare the results. Reuse the query pairs from the conversion step so you exercise the syntax differences directly. For example, to check a numeric range against the sample IMDB movies dataset, run the following query in CloudSearch.

https://my-cloudsearch-domain.us-east-1.cloudsearch.amazonaws.com/2013-01-01/search?q=rating: [7 TO 9]&size=10

Run the equivalent query DSL against your OpenSearch Serverless collection.

GET /imdb_movies/_search
{
  "query": {
    "range": { "rating": { "gte": 7, "lte": 9 } }
  }
}

Confirm that both queries return the same set of movies. Then repeat the comparison for a query that exercises relevance, such as the boosted query from the conversion step, and confirm the top results appear in the same order.

GET /imdb_movies/_search
{
  "query": {
    "bool": {
      "should": [
        { "match": { "title": { "query": "The Matrix", "boost": 6 } } },
        { "match": { "genres": { "query": "Sci-Fi", "boost": 4 } } }
      ]
    }
  }
}

Cut over and operate

When validation passes, update your application to use the OpenSearch Serverless endpoint and the query DSL, and switch from the CloudSearch SDK to the OpenSearch client libraries. After cutover, confirm that no application still points to a CloudSearch endpoint, retain your source data backups in Amazon S3 for rollback, and then delete the CloudSearch domain.

Operating OpenSearch Serverless in production is lighter than operating a domain, because OpenSearch Serverless scales compute for you and you do not tune shards, instance types, or capacity. Your focus shifts to cost and search quality. Monitor OCU consumption and search latency with Amazon CloudWatch, and set alarms on the thresholds that matter to you. Review OCU usage patterns to understand cost and find optimization opportunities, and set capacity limits on the collection to cap the maximum OCUs it can consume. For guidance, see Managing capacity limits for Amazon OpenSearch Serverless and Monitoring Amazon OpenSearch Serverless.

Cost considerations

With OpenSearch Serverless, you pay only for the compute and storage your workload consumes, and OpenSearch Serverless charges for compute and storage separately. OpenSearch Serverless scales indexing compute and search compute independently, so a write-heavy or a read-heavy workload scales only the dimension it needs, and compute can scale to zero when a collection is idle, in which case you pay only for storage. To share hardware across workloads, place collections in a collection group so they draw from the same compute rather than each provisioning its own. For pricing and unit details, see Amazon OpenSearch Service pricing.

Clean up

Because you’re migrating to OpenSearch Serverless, the resources that you’ve created will likely become your production resources. If not, delete any OpenSearch Serverless collections and S3 buckets you created to avoid incurring ongoing cost.

Conclusion

In this post, you saw how Amazon CloudSearch and Amazon OpenSearch Serverless compare, and how the concepts you rely on in CloudSearch (field types, query syntax, autoscaling, and access control) translate into OpenSearch Service. You assess your CloudSearch configuration, model your data with explicit OpenSearch mappings, move your converted documents into the collection with OpenSearch Ingestion, convert your URL-based queries into the OpenSearch query DSL, configure security, and validate before cutover. OpenSearch Serverless gives you the hands-off operational model you have with CloudSearch, and adds richer query capabilities, granular data access policies, and automatic scaling. To get started, create an OpenSearch Serverless collection on the AWS Management Console and follow the steps in this post.

To learn more, see the following resources:

About the authors