Building a Medallion Architecture today typically means that you must build three separate systems working in concert: extract, transform, and load (ETL) jobs to transform data between layers, an orchestrator (such as Apache Airflow or AWS Step Functions) to sequence those jobs in the correct order, and custom change-data-capture (CDC) logic to make sure that each job processes only new or modified records. Each component must be authored, tested, deployed, and maintained independently and when one breaks, the entire pipeline stalls.
In this post, we show how Apache Iceberg materialized views in Amazon SageMaker collapse transformation, orchestration, and incremental processing into a single SQL definition per layer. You declare what each layer should contain, and the system handles when and how it refreshes based on your refresh configuration. With this approach, you can build a Bronze → Silver → Gold pipeline with three SQL statements. This reduces the complexity of maintaining separate orchestration code, CDC logic, and job artifacts.
What is medallion architecture
The medallion architecture organizes data into three progressive layers:
- Bronze layer – Captures raw data as-is from source systems, preserving the original format for auditability and replay.
- Silver layer – Applies cleaning, deduplication, type casting, and business logic to produce validated, query-ready datasets.
- Gold layer – Aggregates Silver data into business-level metrics, key performance indicators (KPIs), and dimensional models optimized for analytics and reporting.
Each layer builds on the previous one, creating clear lineage from raw ingestion to business insight.
Traditional versus declarative approach
The two approaches differ in how much infrastructure you build and maintain.
Traditional approach
You write an ETL job such as Apache Spark script for Bronze to Silver layer and another for Silver to Gold layer. You build a directed acyclic graph (DAG) in Apache Airflow or a Step Functions state machine to run them in order. You implement CDC logic like tracking high watermarks, comparing snapshots, or consuming change streams such that each job processes only new data.
Declarative approach with Iceberg materialized views
You write one CREATE MATERIALIZED VIEW statement per layer with a SCHEDULE REFRESH EVERY N HOURS clause. The AWS Glue managed Spark compute executes the refresh, but you don’t author, version, or deploy a job artifact. Iceberg’s row-level change tracking (position-delete and equality-delete files) identifies which rows changed since the last refresh and AWS Glue processes only those rows. The dependency chain is implicit in the SQL definitions. The only code you maintain is the SQL transformation logic itself.
Apache Iceberg and materialized views
Apache Iceberg is an open-source, high-performance table format designed for petabyte-scale analytic datasets in data lakes. It provides ACID transactions, time travel, schema evolution, and hidden partitioning.
With an Iceberg materialized view, you can define each layer of a medallion architecture as a SQL statement. Under the hood, AWS Glue uses Iceberg’s change-tracking metadata to identify which rows changed since the last refresh, then processes only those rows using managed Spark compute. You configure scheduling and incremental processing through SQL definitions, and the system executes atomic refreshes without requiring you to write pipeline code.
When refreshed, the Gold materialized view reads incrementally from the Silver materialized view, which in turn reads from the Bronze table. This creates a declarative dependency chain: each layer’s definition points to the layer below it, and the system resolves which data to reprocess at each refresh.
Service support for Iceberg materialized views
At time of publication, the following services support creating and refreshing Iceberg materialized views:
For the latest version requirements, see the AWS Glue materialized views documentation.
Technical architecture
The architecture uses Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), as the storage layer. Amazon S3 Tables is a managed Apache Iceberg offering that alleviates the administrative overhead of maintaining Iceberg tables. AWS Glue Data Catalog manages table metadata, and Amazon SageMaker Unified Studio provides the AI-powered notebook environment with AWS Glue 5.1 for authoring and executing materialized view definitions.
The diagram illustrates a three-tier data lakehouse pipeline built on Apache Iceberg. The Bronze layer contains raw trip data (trips_bronze table on S3 Tables with fields: trip_id, city, vehicle_type, fare, status) that you ingest through INSERT/Append operations.
An incremental REFRESH feeds the Silver layer, where a materialized view (mv_trips_silver) performs timestamp conversion, null filtering, and computes derived columns like revenue_per_mile and rating_category. It processes only new or changed rows.
The Silver layer then refreshes two Gold layer materialized views on a daily schedule: mv_city_daily_metrics (city, date, trips, drivers, revenue, tips) and mv_vehicle_performance (vehicle_type, city, trips, revenue, distance). The Gold layer serves downstream consumers including Amazon Athena, Amazon Quick Sight, Amazon Redshift, and first-party (1P) or third-party (3P) compute engines supporting the Iceberg REST API.
The pipeline flows as follows:
Figure 1: The three-tier medallion pipeline from the Bronze table through Silver and Gold materialized views to analytics consumers
Prerequisites
Before starting, verify that you have the following:
- An AWS account with permissions for Amazon SageMaker Unified Studio, AWS Glue, S3 Tables, and AWS Lake Formation.
- An Amazon SageMaker Unified Studio domain.
Step 1: Initialize the environment
Open the AWS Management Console and navigate to Amazon SageMaker.
Choose Get Started to set up Amazon SageMaker Unified Studio.
Choose Open to launch Amazon SageMaker Unified Studio.
After you’re in SageMaker Unified Studio, choose Data in the left pane to create the S3 Tables bucket (a managed Apache Iceberg feature of Amazon S3) and a database. Choose Add, then choose Create S3 Tables Catalog, and provide a catalog and a database name. Finally, choose Create Catalog.
After the catalog creation is complete, in the left navigation pane, choose Notebooks.
Choose Create Notebook.
Before using the notebook, select either Athena Spark or Glue Spark compute connection as the runtime engine for your notebook.
Use the following code samples in individual notebook cells. You can also provide transformation requirements in natural language, and the SageMaker Data Agent will generate SQL code for you.
Add each code block in a new cell by choosing the SQL button:
Choose Athena Spark or Glue Spark as your compute from the cell menu.
If you encounter errors after cell execution, use the data agent chatbot or the Fix with AI button to resolve them.
Step 2: Ingest data into Bronze
Generate 300 realistic ride-sharing trips and insert them directly into the Bronze Iceberg table. This simulates a raw data ingestion layer. In production, you generally configure a streaming source or batch load based on your requirements.
Copy the following code into the first notebook cell (use a Python cell type).
Step 3: Explore Bronze
Run a preview on the bronze table. The output should look like the following screenshot:
You should see raw, unprocessed trip records with string timestamps and nullable fields. This is exactly what the Silver layer will clean up.
Now, verify the ingested data by querying the Bronze table for basic statistics.
The output should look like the following screenshot:
Step 4: Create the Silver materialized view
This SQL statement defines the Silver layer as a materialized view that cleans, transforms, and derives new columns from the Bronze table. Note that this is only a definition. The system processes the data at refresh time.
Verify the Silver layer output:
Notice how the Silver layer now has proper timestamps, derived revenue_per_mile, and rating categories: clean, typed, and ready for you to aggregate.
The output should look like the following screenshot:
Step 5: Create Gold materialized views
Gold materialized views read incrementally from the Silver materialized view. This is a nested materialized view pattern: a materialized view built on top of another materialized view.
Gold 1: City daily metrics
With this materialized view, you can aggregate trip data by city and date with a scheduled daily refresh.
Gold 2: Vehicle performance
With this materialized view, you can aggregate performance metrics by vehicle type and city.
Dependency chain
The complete pipeline dependency is:
Each layer is defined by a single SQL statement. There are no DAGs to maintain, no job definitions to deploy, and no watermark tracking to implement.
Step 6: Query the Gold layer
Query the Gold materialized views to see aggregated business metrics.
City daily metrics Gold table
The output should look like the following screenshot:
Vehicle performance Gold table
The output should look like the following screenshot:
The Gold layer gives you pre-aggregated, business-ready metrics without writing aggregation jobs.
Step 7: Data propagation demo
This section demonstrates how changes propagate through the layers using INSERT, UPDATE (MERGE), and DELETE operations followed by incremental refresh. In production, the scheduled refresh handles this automatically. We trigger it manually here for demonstration purposes.
INSERT new records
Insert new trip records into the Bronze table.
Refresh Silver (incremental)
Refresh the Silver materialized view. Iceberg materialized view processes only three new records.
Verify the new records propagated
The output should look like the following screenshot:
Refresh Gold (cascading from the Silver materialized view)
Refresh the Gold materialized view. It reads from the refreshed Silver materialized view and processes only the incremental changes.
Verify the Gold layer reflects the new trips
The output should look like the following screenshot:
UPDATE through MERGE
Use MERGE to update existing records in Bronze, then refresh incrementally.
Refresh Silver and verify
The output should look like the following screenshot:
Step 8: Cleanup
Drop materialized views, tables, the namespace, and delete the S3 Tables bucket to fully clean up resources.
Limitations and considerations
While materialized views remove most orchestration code, note the following:
- No sub-hour freshness. The minimum schedule granularity is one hour (
SCHEDULE REFRESH EVERY 1 HOUR). - Cascading refresh isn’t automatic. Refreshing Silver doesn’t trigger Gold in the same operation. Each layer refreshes on its own schedule or must be triggered sequentially.
- Deletes require a FULL refresh. An incremental REFRESH that feeds the Silver layer detects inserts and updates through Iceberg metadata but cannot detect row removals. Use
REFRESH ... FULLwhen delete propagation is needed. - SQL subset only. Some window functions, user-defined functions (UDFs), and complex expressions might not be supported in materialized view definitions.
- Schema evolution requires recreation. If the source schema changes in a way that affects the materialized view definition, you must drop and recreate it.
- AWS-specific extension. Iceberg materialized views are not part of the open-source Apache Iceberg specification. They aren’t portable to non-AWS environments.
Pricing
AWS bills materialized view auto-refresh at USD $0.44 per DPU-hour (4 vCPU, 16 GB memory), billed per second with a 1-minute minimum. When you configure scheduled refresh, the AWS Glue Data Catalog uses managed Spark compute to incrementally update the materialized view. You pay only for the compute time of each refresh run.
There are no separate charges for storing materialized view metadata in the Data Catalog (covered under standard catalog pricing: first million objects at no additional cost, then $1.00 per 100K objects/month). The materialized view data itself is stored as Iceberg files in S3 Tables or Amazon S3, charged at standard Amazon S3 storage rates.
Manual refreshes triggered from Spark (through Amazon Athena, Amazon EMR, or AWS Glue notebooks) are billed under those services’ respective compute pricing rather than the materialized view auto-refresh rate. For the latest pricing details, see the AWS Glue pricing page.
Estimated cost for this tutorial: Running through all steps once with 300 records typically consumes less than 0.5 DPU-hours total (~$0.22 in AWS Glue compute plus negligible Amazon S3 storage).
Summary
In this post, you built a Bronze → Silver → Gold medallion architecture using three SQL statements with nested materialized views and no orchestration code. The full pipeline creation took under 2 minutes, and incremental refreshes processed only changed data with no watermarks, no DAGs, no CDC plumbing.
To get started with your own data, create an Amazon SageMaker Unified Studio project, define your Bronze table, and express your transformation logic as Iceberg materialized views. For more information, see the Apache Iceberg materialized views documentation in the AWS Glue Developer Guide.
References
Using materialized views with AWS Glue
Query AWS Glue Data Catalog materialized views
Using materialized views with Amazon EMR
Working with Amazon S3 Tables and table buckets
About the authors




















