Amazon Kinesis Data Streams now supports streaming tables, a fully managed capability that continuously delivers your streaming data as queryable Apache Iceberg tables on Amazon S3 Tables. Amazon S3 Tables is a capability of Amazon Simple Storage Service (Amazon S3). Streaming tables reduce data delivery costs to S3 Tables by up to 50% compared to self-managed alternatives and reduce downstream query costs by up to 30% through intelligent inline compaction that eliminates the small file problem. You need no custom applications, no self-managed compute, and no operational overhead.
Customers increasingly want to unify streaming data with Apache Iceberg for near-real-time analytics, fraud detection, personalization, and artificial intelligence and machine learning (AI/ML) feature pipelines. But integrating the two has meant operating complex custom connectors, managing format conversions, and contending with the performance impact of many small Parquet files that slow queries and increase costs. Streaming tables solve this: configure delivery in a few steps from the console or through APIs, and your data becomes queryable from Amazon Athena, Amazon Redshift, and Apache Spark within minutes. Tables are automatically registered in AWS Glue Data Catalog, making them immediately discoverable for analytics engines and AI agents.
For workloads that don’t require Iceberg table format, you can also deliver streaming data to Amazon S3 general purpose buckets. Delivery is in the source data format, ideal for archival, backup, and ML training data pipelines, with the same serverless, fully managed delivery and no infrastructure to operate.
Challenges with delivering streaming data to Apache Iceberg
Customers today face three challenges when integrating streaming data with Apache Iceberg.
Operational complexity: Connecting Kinesis Data Streams to Iceberg tables today requires deploying and maintaining custom connectors, Apache Flink jobs, or consumer applications. Teams must manage pipeline failures, handle format conversions, scale infrastructure, and monitor delivery reliability. These operational tasks consume significant engineering time and introduce ongoing risk of downtime.
Resiliency and the small file problem: Without proper coordination, simultaneous writes from multiple high-throughput shards can conflict, leading to failed commits, data freshness delays, and degraded performance. Streaming ingestion of high-volume data creates large numbers of small Parquet files in Iceberg tables, forcing a difficult trade-off between data freshness and query efficiency.
Cost: Customers typically spend up to $28/TB operating streaming extract, transform, and load (ETL) pipelines from Kinesis Data Streams using self-managed alternatives based on internal analysis. This creates a high price barrier to getting streaming data into queryable formats and makes cost unpredictable as volume grows.
How delivery to streaming tables solves these challenges
Streaming tables are a native capability built directly into Amazon Kinesis Data Streams. There is no separate service to deploy, no connector to version, and no consumer application to maintain. You enable delivery in a few steps from the console or through APIs.
Zero operational overhead: Streaming tables remove the need to build and operate custom consumer applications for data delivery. No pipeline infrastructure to provision, no scaling logic to write, no failure handling to implement. The capability automatically scales to process gigabytes per second of throughput.
Built-in resiliency: Streaming tables provide write coordination and exactly once delivery semantics across all shards in your stream, resolving concurrent writer conflicts and ensuring data integrity without manual intervention.
Intelligent compaction, no trade-offs: During ingestion, streaming tables perform inline compaction that produces query-optimized Parquet files, eliminating the small file problem while maintaining minute-level data freshness. This reduces downstream query costs by up to 30 percent compared to uncompacted delivery.
Consumption-based pricing: You pay only for data delivered: $14/TB for Iceberg delivery to S3 Tables in US East (N. Virginia) Region (us-east-1) (50% savings compared to self-managed alternatives) and $11/TB for general purpose S3 delivery (60% savings compared to self-managed alternatives). When your stream is idle, you pay nothing for delivery. Combined with Kinesis Data Streams On-Demand Advantage pricing, which eliminates per-shard charges and scales automatically, the entire path from ingestion to queryable Iceberg tables operates on a pure consumption model.
End-to-end managed streaming analytics architecture
With delivery to streaming tables, you now have a fully managed end-to-end real-time data architecture from data ingestion through storage to analytics. Your producers publish events to a Kinesis Data Stream, which continuously delivers data as optimized Iceberg read-only tables in S3 Tables. From there, you can query your streaming data using analytics engines like Amazon Athena, Amazon Redshift, Amazon EMR (Apache Spark), or Apache Flink. You can also let AI agents discover and reason over your data through Glue Data Catalog semantic search. This managed experience removes the intermediate infrastructure that customers previously assembled: separate connector clusters, compaction jobs, and custom consumers. It replaces them with a single, serverless pipeline from stream to insight.
The following diagram illustrates this end-to-end architecture.
Figure 1: End-to-end managed streaming analytics architecture from ingestion to query
Getting started
To get started, sign in to the Amazon Kinesis Data Streams console, navigate to your streams, and enable delivery to streaming tables in a few steps. Specify the stream you want to deliver, configure your schema settings using AWS Glue Schema Registry, and choose your destination S3 Tables location. After you enable it, delivery to streaming tables immediately begins materializing your streaming data as queryable Iceberg tables in S3 with no further intervention required. There’s no infrastructure to provision and no minimum commitment. You pay only for data delivered.
Additionally, you can use Amazon Kinesis Data Streams APIs to programmatically set up, update, or delete delivery to streaming tables configurations for your data streams. With these APIs, teams can build agentic workflows and infrastructure-as-code patterns to manage configurations across multiple data streams at scale.
Getting started with the Kinesis Data Streams Agent Skill
The Kinesis Data Streams Agent Skill provides AI-assisted guidance for setting up streaming tables integrations for your existing or new data streams. The skill helps you configure delivery to S3 Tables (Iceberg) or S3, including schema registry setup, AWS Identity and Access Management (IAM) role configuration, and validation.
Installing as an Agent Skill
Agent Skills are discovered automatically by compatible tools through the SKILL.md file. Refer to the Agent Toolkit for AWS Skill Installation Guide to install the managing-amazon-kinesis-data-streams Agent Skill. We also recommend you install the AWS MCP Server in your developer tool of choice, which exposes tools for searching AWS documentation, blogs, and Skills dynamically at runtime. These capabilities make agents more accurate and powerful for AWS related development and operational tasks, and make skill discovery and installation more flexible. Refer to Setting up the AWS MCP Server for guidance on installing the AWS MCP Server in your environment.
For example:
To verify the installation, interact with the skill in your preferred tool.
To start delivering data from your data streams to Apache Iceberg tables in real time, prompt “Create me a streaming table on my events data stream” to your agent of choice:
Figure 2: Prompting the agent to create a streaming table
The agent dynamically loads the managing-amazon-kinesis-data-streams skill and starts by gathering the available resources in your AWS account for the streaming tables integration. After it gathers that data, it confirms the resources to use or create, and creates the integration:
Figure 3: The agent confirming resources before creating the integration
After creating the integration, the agent summarizes the status and can then help with any other operational tasks with your data. For example, the agent can help you set up AWS Lake Formation permissions to query the data in S3 Tables with Athena, or configure your table maintenance behavior in S3 Tables:
Figure 4: The agent offering follow-up operational tasks
Conclusion
Streaming tables are available in all AWS Regions where Amazon Kinesis Data Streams is offered. Pricing is $14/TB for delivery to S3 Tables (Apache Iceberg) and $11/TB for delivery to general purpose S3 buckets. To learn more, visit the documentation and pricing pages.
About the authors
