OpenSearch is an open source software suite for search, analytics, security monitoring, and observability applications, licensed under the Apache License V2.0. Amazon OpenSearch Service is a managed service that lets you deploy, scale, and operate OpenSearch and the Elasticsearch engine in the AWS Cloud. Customers run search workloads on OpenSearch Service at a scale of billions of documents. When a single index holds millions to billions of documents, you need to plan the topology of the OpenSearch Service domain that holds the index. This post walks through how Autodesk re-architected a single-index Elasticsearch 7.1.1 domain on Amazon OpenSearch Service into four multi-index OpenSearch Service domains, using Migration Assistant for Amazon OpenSearch Service and a routing layer that directs each query to the shards that hold the data for that query.
Autodesk is a technology company that serves customers across three industry verticals: Architecture, Engineering, and Construction (AEC), Product Design and Manufacturing, and Media and Entertainment. Autodesk’s mission is to empower everyone, everywhere to design and make anything, helping customers work across the boundaries of project, discipline, and industry.
Autodesk Forma (formerly Autodesk Construction Cloud, or ACC) is a cloud-based construction management and collaboration system. Customers across the globe use Autodesk Forma for workflows that include document management, bid management, quantification, coordination, design collaboration, project management and field collaboration. Autodesk Forma uses Amazon OpenSearch Service to provide a search experience for millions of users. As customers add data, the data that Forma stores in OpenSearch Service grows. In an OpenSearch Service domain, an index is the unit of data storage and organization. When an index reaches 100 TB, the index becomes a performance bottleneck and is hard to scale. As Autodesk Forma grew, Forma data management (formerly Autodesk Docs) hit performance and scaling limits. This component supports access and search across the project catalog.
Where Autodesk started
Forma data management ran on a single Elasticsearch 7.1.1 domain on Amazon OpenSearch Service with one index. The domain held about 100 TB of data on over 100 data nodes with over 400 primary shards and a replication factor of 1. The average shard held 200 GB. Because of the scale and the production state of the domain, tuning techniques such as adding shards, adding indices, or rebalancing data were not viable.
The single-index, single-domain design exposed three challenges to future data growth:
- Query performance. Query latency degraded over time as data grew.
- Vertical scaling. The team had reached the limit of the largest Amazon Elastic Compute Cloud (Amazon EC2) instance size available for the existing instance class.
- Horizontal scaling. Without a routing mechanism, adding nodes produced hot nodes inside the cluster managed by the OpenSearch Service domain.
Multi-domain architecture with intelligent routing
Vertical or horizontal scaling can address query performance in the short term, but neither addresses the underlying single-index, single-domain scalability limit. A horizontal scaling approach that uses routing keys gives you control over which shards each query touches, without requiring larger hardware. The Autodesk team applied this approach to re-architect the search service without impacting production traffic.
Figure 1: Multi-domain architecture with intelligent routing
The architecture has the following properties:
- Four Amazon OpenSearch Service domains on OpenSearch 2.19, each running 24
m7i.4xlarge.searchnodes. - 24 indices total (6 per domain).
- About 95 million documents per index.
- 52 TB of primary storage. This is 37 percent smaller than the primary storage size of the original single-index domain, mainly because the migration skipped deleted documents.
The setup uses four horizontally scaled OpenSearch Service domains, with a routing layer that directs each query to the domain that holds the project’s data.
The architecture uses a Amazon DynamoDB table that stores 4.3 million routing records, one record per project. A project is the primary workspace in Forma data management, where teams, data, documents, models, workflows, permissions, issues, and collaboration activities live together. Forma application looks up the Amazon DynamoDB table for the project-to-domain mapping and then issues the search query to the correct domain.
Redistributing millions of records across four domains was hard. To find an even project-to-index allocation, the team used a bin-packing algorithm. A bin-packing algorithm packs items of varying sizes into a fixed number of bins to minimize waste and produce an even distribution. The team worked with 4.3 million projects of varying document counts, from a few documents per project up to millions, across 24 indices that each target around 400 million documents. The team implemented a stratified bin-packing algorithm that uses historical usage metrics for the workload. This algorithm avoids over- or under-allocation of resources during migration planning. To avoid over-allocation, the team used the 95th percentile (P95) usage metric. After applying the algorithm, each OpenSearch Service domain landed at about 49 percent utilization, which leaves a 2x growth buffer. The application then uses routing-key-based queries to search only the relevant shards, instead of every shard in the index.
The architecture has the following benefits:
- Horizontal scalability. The team can add more domains and indices as needed.
- Efficient routing. Queries hit specific shards, not every shard in the domain.
- Reduced blast radius. If one domain becomes unavailable, only about 25% of traffic is affected, instead of full downtime under the single-domain design.
- Independent scaling. The team can scale each domain based on its load pattern.
- More search threads. The aggregate search-thread pool is larger across four domains than on one domain.
Migration steps
The following sections describe the four steps the Autodesk team followed to complete the migration.
Step 1: Categorize projects by size
The team grouped projects into four size categories by current document count, then collected data over six months to compute a per-category growth factor and extrapolate one year out:
| Category | Document range | Project count | % of total | P95 growth factor | Rationale |
| TINY | 0 – 1,000 | 4,085,310 | 95.0% | 3.82x | Tiny projects grow fastest |
| SMALL | 1,000 – 10,000 | 184,347 | 4.3% | 2.11x | Moderate growth expected |
| MEDIUM | 10,000 – 100,000 | 28,385 | 0.66% | 1.72x | Slower relative growth |
| LARGE | 100,000+ | 2,266 | 0.05% | 1.38x | Already mature, minimal growth |
| Total | — | 4,300,308 | 100% | — | — |
The table shows that 95 percent of projects are TINY, but LARGE projects account for the bulk of document volume. The stratification by category lets the algorithm handle each category appropriately.
The Autodesk team analyzed document count per project over six months to estimate growth. Using the P95 growth factor per category gives a conservative capacity plan that covers 95 percent of projects and avoids over-provisioning.
Step 2: Interleaved distribution
If you process all LARGE projects first, you create imbalance across the indices. To avoid this imbalance, the bin-packing algorithm interleaves the categories in a round-robin pattern. The team used the following sequence to distribute documents evenly across the Amazon OpenSearch Service domains:
- Sort the projects within each category, largest first.
- Create a queue for each category. The queue is a first-in, first-out data structure that holds the sorted projects for one category.
- Distribute projects in a round-robin pattern: pick one from LARGE, then MEDIUM, then SMALL, then TINY, and repeat.
Step 3: Load-balanced best fit
After interleaving, the team computed the projected size of each project and assigned the project to an index. The following steps describe the approach:
- Compute the estimated future size as
current size × growth factor. - Use a priority queue to find the index with the most available capacity. In a priority queue, each element has a priority. Here, the priority of each index is the amount of available capacity the index has. Unlike a regular queue, a priority queue returns the highest-priority element first, not the first one inserted.
- Assign the project to the index that has the most available capacity.
- Update the index’s estimated load and re-insert the index into the priority queue with the new capacity. The re-insert step keeps the queue accurate for the next project assignment.
The preceding three steps produced the following results:
- The algorithm distributed 4.3 million projects with 99.999 percent routing accuracy.
- Project distribution across indices held to a 0.15 percent variance.
- Each domain landed at 49.1 percent capacity utilization after applying growth factors, leaving 50.9 percent headroom for future growth.
- The algorithm computed the 4.3 million project allocations in about 10 minutes.
The team stored the project-to-index allocation mapping in Amazon DynamoDB for real-time query routing. Routing controls how the application uses domain resources and how each domain performs. With routing, the application searches the shards that match the routing key (projectId) for that project. Without routing, the same query searches every shard in the index, which wastes domain resources and produces slower queries. The team also tuned the shard size, which matters most for large projects. One of the largest projects held 7 million documents at about 40 KB per document, for a total of about 280 GB. To split the data for that project into 20–25 GB shards, the team set routing_partition_size to 12.
Step 4: Migration with Migration Assistant for Amazon OpenSearch Service
The Autodesk team used the snapshot and re-index path in Migration Assistant for Amazon OpenSearch Service to migrate 2.3 billion documents. Migration Assistant for Amazon OpenSearch Service adapts to the migration profile and provides AWS Identity and Access Management (IAM) permission boundaries, Amazon Virtual Private Cloud (Amazon VPC) support, and the security policies the migration needs. Migration Assistant for Amazon OpenSearch Service integrated with the over 400 tasks that run the application on Amazon Elastic Container Service (Amazon ECS) with AWS Fargate.
Before the production cutover, the team ran several proof-of-concept (PoC) iterations and tuned the migration configuration to raise throughput from 18 GB/hr to 228 GB/hr. The first PoC iteration hit 18 GB/hr on m7g.large.search nodes. Each subsequent iteration added horizontal scale, larger instances (m7g.2xlarge.search and m7g.4xlarge.search), parallel writes across domains, and zero replicas during migration. The fourth and final PoC iteration hit 228 GB/hr. Multiple PoC iterations helped the team select the optimal instance size and instance class to migrate 2.3 billion documents in 6 hours with zero downtime and no customer incidents.
Post-migration analysis
After the team migrated 2.3 billion documents with routing enabled, the shards landed as follows:
| Metric | Result | Target | Status |
| Total primary shards | 4,325 | — | ✓ |
| Total data size | 52.11 TB | ~52 TB | ✓ On target |
| Average shard size | 12.34 GB | 10–15 GB | ✓ Optimal |
| Median shard size | 11.9 GB | 10–15 GB | ✓ Optimal |
| Shards in optimal range (10–15 GB) | 75.5% | 70% | ✓ Above target |
| Hot shards (> 30 GB) | 12 (0.28%) | < 1% | ✓ Within limit |
| Undersized shards (< 10 GB) | 528 (12.2%) | < 15% | ✓ Within limit |
| Cross-domain balance | 2.3% variance | < 5% | ✓ Within target |
| Node balance (StdDev) | 0.78–1.12 shards | < 2 | ✓ Within target |
The following table compares the pre- and post-migration architectures:
| Aspect | Old (single domain) | New (four domains with intelligent routing) |
| Shard size | 200 GB average | 12.34 GB average (94% reduction) |
| Query broadcast | All 400+ shards | ~12 shards (97% reduction) |
| Shards in optimal range | 0% | 75.5% |
| Cross-domain balance | N/A (single domain) | 2.3% variance |
| Storage | 83.3 TB | 52 TB |
| Total P99 query latency | 17 seconds | 5 seconds |
The team migrated 2.3 billion documents in about 6 hours. Storage dropped by about 37 percent, from 83.3 TB to 52 TB, because the migration dropped deleted documents. The migration produced 4,325 shards at an average of 12.34 GB per shard, distributed across the four domains. 75.5 percent of shards landed in the 10–15 GB range, compared to 210 GB before the migration, which confirms that the new architecture solves the large-shard problem. The shard size is as per general guidance where search latency is a key performance objective. Cross-domain variance of 2.3 percent (12.85 TB to 13.15 TB per domain) confirms even data distribution.
After the migration, queries that include the projectId routing key scan only the relevant shards (typically 12 of 180 per index), which reduces search load across shards by 93 percent. Routing also balances CPU and memory use across each domain. The routing_partition_size of 12 per index produced the right shard count per index. Overall P99 latency improved by 72 percent, from 17 seconds to 5 seconds. Within that figure, search-query P99 improved by 92 percent, from 2,500 ms to 200 ms.
Lessons learned
The PoC iterations surfaced several lessons. Larger instance types help query performance in the short term, but query routing combined with horizontal scaling produces higher sustained throughput. During bulk loads, disable replicas and increase the refresh interval to reduce write overhead. Plan for enough IP addresses and subnet capacity when you scale the application out, so that you do not hit a service limit mid-migration. Validate the VPC routing configuration between the application and the OpenSearch Service domains. Confirm OpenSearch Service data-node capacity with AWS Support before a horizontal scale-out. The Amazon DynamoDB-based routing layer adds about 20 ms of routing latency per query, but the routing layer cuts overall search latency and unlocks horizontal scale.
Conclusion
In this post, you saw how the Autodesk team migrated 2.3 billion documents from a single-index domain to four multi-index Amazon OpenSearch Service domains in about 6 hours.
Transitioning to a multi-domain architecture or updating to the latest OpenSearch version has historically been complex. It can also be difficult to predict the outcome of a migration before production traffic moves. The Migration Assistant for Amazon OpenSearch Service solution addresses these challenges by making migration workflow-driven, repeatable, and more straightforward to validate before cutover.
Migration Assistant for Amazon OpenSearch Service coupled with Amazon DynamoDB-based intelligent routing helped achieve balanced shards and improved search query performance. Multiple PoC iterations helped find routing bugs, service-quota limitations, and infrastructure-provisioning gaps before the production cutover.
If you plan to migrate a large dataset between OpenSearch Service domains, you can use Migration Assistant for Amazon OpenSearch Service. For more information, see the Migration Assistant for Amazon OpenSearch Service documentation.
About the authors
