Build with geospatial and variant types in Iceberg v3 on AWS Glue 6.0

As organizations build data lakes that combine geospatial data, high-frequency event streams, and heterogeneous payloads, the limitations of older table formats become acute. Without a native geospatial type, coordinates require separate float columns (latitude/longitude) with no spatial predicates. Without nanosecond-precision timestamps, sub-microsecond event ordering is lost. Without a variant type, semi-structured data forces a choice between rigid flattening and untyped JSON strings. Each workaround adds complexity, slows queries, and increases maintenance burden.

AWS Glue 6.0, powered by Apache Spark 4.1, removes these workarounds by adding support for Apache Iceberg v3, bringing new column-level capabilities to your data lake tables. These include new data types: native geospatial types (GEOMETRY with spatial predicates, and GEOGRAPHY), nanosecond-precision timestamps, and the VARIANT type for semi-structured data with automatic shredding. Iceberg v3 also adds support for DEFAULT column values. These are table format features. After they’re written, they’re readable by any Iceberg v3-compatible engine that supports these features.

In this post, we build a connected vehicle fleet monitoring pipeline that uses these capabilities in a single Iceberg v3 table. Vehicles emit telemetry events with GPS coordinates (geospatial), sub-microsecond event times (nanosecond), and sensor payloads that vary by vehicle type (variant). We ingest these events, run spatial queries to detect geofence violations, sequence events at nanosecond precision, and extract typed metrics from heterogeneous payloads, all without workarounds, flattening, or external libraries.

Solution overview

A logistics company operates a mixed fleet of delivery vehicles: vans, electric bikes, and delivery robots. Each vehicle type produces telemetry events with a different sensor payload schema. The operations team needs to:

1. Detect geofence violations: flag vehicles that enter restricted zones (airports, pedestrian areas, private property).
2. Sequence events precisely: at fleet scale, many events land in the same microsecond window. Nanosecond timestamps give a deterministic order and prevent ties when sequencing or deduplicating events during processing.
3. Extract metrics from heterogeneous payloads: query battery level from delivery robots, fuel level from vans, and pedal cadence from bikes, all stored in the same column.

We address all three requirements with a single Iceberg v3 table on AWS Glue 6.0. The following data definition language (DDL) shows the table structure. The AWS Glue job we provision in subsequent steps executes this statement.

CREATE TABLE fleet_monitoring_db.vehicle_telemetry (
event_id STRING,
vehicle_id STRING,
vehicle_type STRING DEFAULT 'UNKNOWN',
event_time TIMESTAMP_NTZ(9),
location GEOMETRY(4326),
service_area GEOGRAPHY(4326),
sensor_payload VARIANT,
speed_kmh DOUBLE DEFAULT 0.0,
region STRING DEFAULT 'EMEA'
) USING ICEBERG
TBLPROPERTIES (
'format-version' = '3',
'write.delete.mode' = 'merge-on-read'
)
PARTITIONED BY (days(event_time), vehicle_type)

In the preceding statement, the database is shown as fleet_monitoring_db for readability. The deployed stack creates it as fleet_monitoring_<account-id>.

The following list describes the key columns:

• event_time TIMESTAMP_NTZ(9): Stores the event timestamp at nanosecond precision.
• location GEOMETRY(4326): Stores GPS coordinates as native spatial objects using (SRID 4326). You can use predicates like ST_Intersects directly in SQL, replacing hand-coded spatial math on raw latitude/longitude doubles (WGS 84).
• service_area GEOGRAPHY(4326): Stores geographic coordinates using a spherical (geodesic) model, distinct from GEOMETRY’s planar model. AWS Glue 6.0 writes and reads GEOGRAPHY in Iceberg v3, and the type is portable to any Iceberg v3-compatible engine. Geodesic spatial predicates over GEOGRAPHY are engine-dependent today. In this post we run spatial queries on the GEOMETRY location column, which Glue 6.0 supports natively.
• sensor_payload VARIANT: Each vehicle type produces a different JSON schema. Vans report fuel and engine metrics, robots report battery and camera status, bikes report cadence and heart rate. All land in this single column without schema unions or separate tables using variant data type.
• vehicle_type STRING DEFAULT ‘UNKNOWN’ and speed_kmh DOUBLE DEFAULT 0.0: When an ingestion writer omits these fields, Iceberg applies the declared defaults automatically. Useful when multiple producers write to the same table and not all of them populate every column.

The table uses PARTITIONED BY (days(event_time), vehicle_type) so that analytical queries can prune by date range and vehicle type without scanning the full table. 'write.delete.mode' = 'merge-on-read' supports fast row-level corrections (for example, correcting a misreported GPS coordinate) through compact deletion vectors (Roaring Bitmaps) instead of accumulating positional delete files.

In this post, we insert sample data directly to focus on the new Iceberg data types and how to use them together. In production, these events would stream from Amazon Managed Streaming for Apache Kafka (Amazon MSK) into an AWS Glue 6.0 streaming job.

The following diagram illustrates the production architecture for reference:

Figure 1: Reference architecture for a fleet telemetry pipeline on AWS Glue 6.0

The architecture processes vehicle telemetry through two paths, with a downstream batch analytics layer:

Hot path (real-time, milliseconds): A Spark Real-Time Mode (RTM) job reads telemetry from Amazon MSK and evaluates geofence violations using spatial predicates like ST_Intersects, routing alerts to a downstream Kafka topic within milliseconds.

Cold path (near-real-time, seconds): A micro-batch job reads the same MSK topic and writes events into an Iceberg v3 table, converting payloads to GEOMETRY, TIMESTAMP_NTZ(9), and VARIANT columns with DEFAULT values applied.

Batch analytics: An AWS Glue job reads the Iceberg v3 table to run batch analytics on geofence detection, nanosecond event sequencing, and per-vehicle-type metric extraction.

Prerequisites

To follow along, you need:

• An AWS account and an AWS Region where AWS Glue 6.0 is available.
• An AWS Identity and Access Management (IAM) role with permissions to deploy AWS CloudFormation stacks and create resources including AWS Glue, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch Logs.

Deploy the CloudFormation stack

We provide an AWS CloudFormation template that provisions all the resources needed for this walkthrough.

The stack provisions the following resources:

• An Amazon S3 bucket for Iceberg table storage.
• An IAM role with permissions for AWS Glue, Amazon S3, and Amazon CloudWatch Logs.
• An AWS Glue database (fleet_monitoring_<account-id>).
• An AWS Glue job fleet-telemetry-ingest-<account-id> (PySpark): creates the Iceberg v3 table vehicle_telemetry described earlier and inserts sample telemetry from three vehicle types.
• An AWS Glue job fleet-telemetry-queries-<account-id> (PySpark): demonstrates geofence detection, nanosecond sequencing, variant extraction, and default values.

Deploy the CloudFormation stack:

1. Download the CloudFormation template from the GitHub repository.
2. Sign in to the AWS CloudFormation console.
3. Choose Create stack, With new resources, Upload a template file, and upload the downloaded template.
4. Acknowledge the IAM capabilities and choose Create stack.

Stack creation takes approximately 2–5 minutes. No parameters are required.

After the stack completes, navigate to the AWS Glue console and run the jobs in this order:

1. Run fleet-telemetry-ingest-<account-id>. This job creates the Iceberg v3 table and inserts sample data (approximately 2 minutes).
2. After it succeeds, run fleet-telemetry-queries-<account-id>. This job executes all demonstration queries (approximately 2 minutes).

The following sections describe each job in detail.

Job 1: Ingest sample telemetry data

The ingestion job creates the Iceberg v3 table described earlier and inserts four sample telemetry events: one for each of the three vehicle types (van, robot, bike), plus one with omitted fields to demonstrate DEFAULT values. You can view the complete script in the GitHub repository. Note that the geospatial types require one additional Spark configuration (spark.sql.geospatial.enabled=true), which is already set in the job’s --conf argument by the CloudFormation template. All other types work with no extra configuration.

The following are the key snippets from the script:

Van telemetry: GPS coordinates with engine metrics and route information:

spark.sql(f"""
INSERT INTO {TABLE} VALUES (
'EVT-001', 'VAN-042', 'VAN',
CAST('2026-07-28 09:15:30.123456789' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'0101000000E17A14AE47E1C0BF1F85EB51B84E4940'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'0101000000E17A14AE47E1C0BF1F85EB51B84E4940'), 4326),
PARSE_JSON('{{"fuel_pct": 0.72, "cargo_kg": 450, "door_open": false,
"engine": {{"rpm": 2100, "temp_c": 88.5}},
"route": {{"stops_remaining": 4, "eta_minutes": 35}}}}'),
35.2, 'EMEA'
)
""")

Delivery robot telemetry: Same table, completely different sensor schema (battery, cameras, navigation):

spark.sql(f"""
INSERT INTO {TABLE} VALUES (
'EVT-002', 'ROB-117', 'ROBOT',
CAST('2026-07-28 09:15:30.123456790' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'01010000000000000000001040000000000000F03F'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'01010000000000000000001040000000000000F03F'), 4326),
PARSE_JSON('{{"battery_pct": 0.62, "obstacle_distance_m": 2.8,
"navigation_mode": "autonomous",
"cameras": {{"front": "active", "rear": "recording"}}}}'),
48.0, 'EMEA'
)
""")

Note: EVT-001 and EVT-002 are exactly 1 nanosecond apart (.123456789 vs .123456790). Without TIMESTAMP_NTZ(9), both would round to the same microsecond and be indistinguishable.

Default values test: Event inserted with vehicle_type, speed_kmh, and region omitted:

spark.sql(f"""
INSERT INTO {TABLE}
(event_id, vehicle_id, event_time, location, service_area, sensor_payload)
VALUES (
'EVT-004', 'UNK-999',
CAST('2026-07-28 10:00:00.000000000' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'0101000000000000000000F03F000000000000F03F'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'0101000000000000000000F03F000000000000F03F'), 4326),
PARSE_JSON('{{"status": "initializing"}}')
)
""")

The omitted columns automatically receive their DEFAULT values: vehicle_type="UNKNOWN", speed_kmh = 0.0, region = 'EMEA'.

Job 2: Query the data

The query job demonstrates all four data types working together. After the job succeeds, select the run in the AWS Glue console and choose Output logs to see the results.

The following sections walk through the key queries from the job and the results of each.

Geofence detection with ST_Intersects

The job defines a polygon and finds all vehicles inside it:

POLY = "010300...."
SELECT event_id, vehicle_id, vehicle_type, speed_kmh
FROM fleet_monitoring_db.vehicle_telemetry
WHERE ST_Intersects(
location,ST_SetSrid(ST_GeomFromWKB(X'{POLY}'), 4326)
)
ORDER BY event_id

The polygon covers coordinates (0,0)-(5,0)-(5,2)-(0,2). Three vehicles are inside (ROBOT at (4,1), BIKE at (3,1), UNKNOWN at (1,1)). The VAN at (-0.1278, 51.5074) is outside.

Figure 2: Geofence query results showing the three vehicles inside the polygon

Nanosecond event sequencing

Order events by their sub-microsecond timestamps:

SELECT event_id, vehicle_id, CAST(event_time AS STRING) AS precise_time
FROM fleet_monitoring_db.vehicle_telemetry
WHERE event_id IN ('EVT-001', 'EVT-002', 'EVT-003')
ORDER BY event_time ASC

EVT-001 and EVT-002 are correctly distinguished and ordered despite being only 1 nanosecond apart. With standard TIMESTAMP_NTZ (microsecond precision), both would show .123456 and their relative order would be undefined.

Figure 3: Nanosecond-precision ordering distinguishing two events one nanosecond apart

Different sensor schemas per vehicle type, all extracted with variant_get:

SELECT vehicle_id, vehicle_type,
CASE vehicle_type
WHEN 'VAN' THEN variant_get(sensor_payload, '$.fuel_pct', 'DOUBLE')
WHEN 'ROBOT' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
WHEN 'BIKE' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
ELSE NULL
END AS energy_level,
variant_get(sensor_payload, '$.engine.temp_c', 'DOUBLE') AS engine_temp,
variant_get(sensor_payload, '$.cameras.front', 'STRING') AS front_cam,
variant_get(sensor_payload, '$.deliveries.completed', 'INT') AS deliveries_done
FROM fleet_monitoring_db.vehicle_telemetry
WHERE vehicle_type != 'UNKNOWN'
ORDER BY vehicle_id

Figure 4: Variant extraction returning typed values from heterogeneous sensor payloads

variant_get takes three arguments: the column, a dot-path expression, and the expected return type. It supports arbitrary nesting depth. $.engine.temp_c reaches two levels deep, $.deliveries.completed reaches into a different structure entirely. When a path doesn’t exist in a particular row’s payload, it returns NULL.

Default values

Confirm that omitted columns received their defaults:

SELECT event_id, vehicle_type, speed_kmh, region
FROM fleet_monitoring_db.vehicle_telemetry
WHERE event_id = 'EVT-004'

Figure 5: Default column values applied to the event inserted with omitted fields

EVT-004 was inserted without vehicle_type, speed_kmh, or region. The declared defaults were applied automatically.

Combined query: Combining spatial, temporal, and variant operations

The following query runs a geospatial predicate, nanosecond ordering, and variant extraction in a single SELECT statement:

SELECT vehicle_id, vehicle_type,
CAST(event_time AS STRING) AS precise_time,
CASE vehicle_type
WHEN 'VAN' THEN variant_get(sensor_payload, '$.fuel_pct', 'DOUBLE')
WHEN 'ROBOT' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
WHEN 'BIKE' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
ELSE NULL
END AS energy_level,
speed_kmh
FROM fleet_monitoring_db.vehicle_telemetry
WHERE ST_Intersects(location, ST_SetSrid(ST_GeomFromWKB(X'0103000000...'), 4326))
ORDER BY event_time ASC

Figure 6: Combined query results over a single Iceberg v3 table

This single query combines a spatial predicate, nanosecond ordering, and variant extraction over one table, with no external libraries, pre-processing, or joins to separate geometry or payload tables.

Clean up

To avoid ongoing charges from the AWS Glue jobs and Amazon S3 storage, delete the CloudFormation stack when you’re done:

1. Open the AWS CloudFormation console.
2. Select the stack you deployed earlier and choose Delete.

Conclusion

In this post, we stored and analyzed geospatial coordinates, nanosecond timestamps, and heterogeneous sensor payloads in a single Iceberg v3 table on AWS Glue 6.0, with sensible defaults applied automatically, no external libraries, and no schema flattening.

• GEOMETRY columns replace latitude/longitude doubles and support native spatial predicates like ST_Intersects for geofence detection. GEOGRAPHY is stored natively.
• TIMESTAMP_NTZ(9) preserves full nanosecond precision for event sequencing where microsecond resolution is insufficient.
• VARIANT stores heterogeneous payloads (different schema per vehicle type) in one column with typed extraction through variant_get.
• DEFAULT values keep field population consistent across multiple ingestion writers without duplicating logic.

All capabilities require Iceberg format-version 3. Geospatial requires one additional configuration (spark.sql.geospatial.enabled=true). Nanosecond timestamps, Variant, and DEFAULT values work with no extra configuration.

These capabilities apply wherever schemas vary by source (IoT fleets, multi-tenant software as a service (SaaS), event-driven architectures), timestamps need sub-microsecond precision (trading, sensor fusion, autonomous systems), or spatial operations replace coordinate workarounds (logistics, real estate, delivery networks).

For more information, see the AWS launch announcement (launch URL to be added before publishing), the AWS Glue documentation, and the Apache Iceberg v3 specification. AWS Glue 6.0 includes additional capabilities such as Spark Real-Time Mode and Spark Declarative Pipelines, which we cover in separate posts.

About the authors