Automate Spark Scala migration to 4.x with AWS Spark Upgrade Agent

If you’re a data worker responsible for managing Apache Spark 3.x workloads on Amazon EMR before, you’ve likely faced the challenge of migrating hundreds of jobs to Spark 4.0 without disrupting production pipelines. In this post, you will learn how to automate Spark 3.x to 4.0 migration using the AWS Spark Upgrade Agent covering API deprecations, behavioral changes, build configuration updates, and job validation. What once took months of manual effort can be completed in hours.

This is part 3 of a three-part series on how the AWS Spark Upgrade Agent can automate and simplify Spark upgrades.

Part 1 introduces the agent’s architecture and capabilities. Part 2 walks through a complete PySpark migration from Spark 3.5 to Spark 4.0 on Amazon EMR Serverless. This post walks through Scala migration from Spark 3.3 (Scala 2.12) to Spark 4.0 (Scala 2.13).

Apache Spark 4.0 on Amazon EMR 8.x delivers improvements like native merge_into() support, enhanced Adaptive Query Execution, improved Python UDF performance through Arrow-based serialization, and major Structured Streaming enhancements. For teams on Spark 2.4 or 3.x, the complexity lies in managing API deprecations, behavioral changes, build configuration updates, and re-validating hundreds of jobs while maintaining production pipelines.

Prerequisites

This post assumes you’ve completed the one-time AWS CloudFormation setup and proxy configuration detailed in the introduction post.

What is the Spark Upgrade Agent?

The AWS Spark Upgrade Agent is a fully managed remote server that automates Spark migration using a Model Context Protocol (MCP) interface for code analysis and transformation. For details, see the introduction post.

Architecture: How it works

The architecture follows a least-privilege security model:

• Scoped IAM roles — AWS IAM roles are scoped to only MCP server calls, Amazon Simple Storage Service (Amazon S3) staging bucket access, and Amazon EMR job submission.
• Local source code — Your source code stays local, with only minimal diagnostic information transmitted.
• Encryption in transit — All data is encrypted in transit.
• Audit trail — AWS CloudTrail records every tool invocation for full auditability.

Example use case: Enterprise-scale Spark migration

To illustrate the capabilities of the agent at scale, consider a real-world migration scenario from a large company. This company runs a data processing platform with thousands of Spark jobs across Scala, PySpark, and Spark SQL workloads, with a code base spanning Spark 3.3 and 3.5.

The data engineering team faces a migration across three workload types with different complexity profiles:

• Spark SQL applications: The most portable, but still requiring validation of behavioral changes in the query optimizer and join strategies introduced in Spark 4.0.
• PySpark workloads: Requiring updates to UDF serialization patterns, Arrow-based optimizations, and DataFrame API changes.
• Scala applications: The most complex, involving build system updates (Maven and SBT), API deprecations, and recompilation against new Spark 4.0 JARs.

To tackle this, the team uses the Spark Upgrade Agent to migrate all three workload types to Spark 4.0 on Amazon EMR 8.x. For each workload, the agent is invoked directly from Kiro or VS Code with Cline, applying targeted transformations and immediately validating results against a live Amazon EMR 8.x Serverless application running Spark 4.0.

Migrations that would traditionally require months of manual engineering effort complete in a fraction of the time. The agent follows an iterative refinement loop: it performs local validation, then submits the job to a remote Amazon EMR cluster. This loop catches and resolves runtime failures automatically, reducing the need for manual debugging cycles. Build configuration files (pom.xml and build.sbt) are updated automatically by the agent, eliminating a common source of migration errors.

Solution walkthrough

The following sections walk through the complete migration workflow, from initial setup through advanced code transformations and validation.

1. Setup

This section lists what you need before starting. Some items, such as Amazon EMR Serverless applications, can be created during the walkthrough using the agent if they don’t already exist.

Must have before starting:

• AWS Command Line Interface (AWS CLI) configuration: Your AWS CLI must be configured with a profile that has the necessary permissions. See Configuring the AWS CLI for details.
• IAM role with Amazon EMR permissions: An AWS CloudFormation template is provided in the setup guide to provision the required IAM role. The role is scoped to the permissions needed for the upgrade process: calling the MCP server, reading and writing to the Amazon S3 staging bucket, and submitting Amazon EMR jobs.
• Amazon S3 staging bucket for artifacts: Used to store code artifacts and Amazon EMR job outputs during the validation phase.
• Integrated development environment (IDE) installation: Kiro or VS Code with MCP support (Cline extension). Either IDE can interact with the Spark Upgrade Agent through natural language prompts. Consult the setup guide for Kiro and Cline’s documentation to use the MCP server with Cline.
• One-click MCP server installation: The dataprocessing-mcp server is installed and configured as described in the setup guide.
• Amazon EMR Serverless applications: An Amazon EMR Serverless application is required for the validation workflow:
  • Target application (Spark 4.0): An Amazon EMR Serverless application configured with release label emr-spark-8.0.0, used to validate migrated jobs against Spark 4.0 on Amazon EMR 8.0.

1.1 Infrastructure setup (AWS CloudFormation)

Two AWS CloudFormation stacks create the required resources: an AWS IAM role, an Amazon S3 staging bucket, an Amazon EMR Serverless application (Spark 4.0), and its execution role.

Stack 1: AWS IAM role and Amazon S3 staging bucket

The spark-upgrade-mcp-setup template creates the AWS IAM role and Amazon S3 staging bucket required by the upgrade agent. Choose the Launch Stack button for your Region. For additional Regions, see the full Region list.

After deployment, open the AWS CloudFormation Outputs tab, copy the ExportCommand value, and run it in your terminal. This sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH automatically.

The following figure shows the Outputs tab with the ExportCommand value.

Outputs tab of the AWS CloudFormation stack showing the ExportCommand value

# Sets SMUS_MCP_REGION, IAM_ROLE, and STAGING_BUCKET_PATH
export SMUS_MCP_REGION=<YOUR-REGION> && export IAM_ROLE=arn:aws:iam::<YOUR-ACCOUNT-ID>:role/spark-upgrade-role-* && export STAGING_BUCKET_PATH=<amzn-s3-demo-bucket>

Then configure the AWS CLI profile:

aws configure set profile.spark-upgrade-profile.role_arn ${IAM_ROLE}
aws configure set profile.spark-upgrade-profile.source_profile default
aws configure set profile.spark-upgrade-profile.region ${SMUS_MCP_REGION}

Stack 2: Amazon EMR Serverless target application and execution role

The emr-serverless-target-setup template creates an Amazon EMR Serverless application configured with Spark 4.0 (release label emr-spark-8.0.0) and a shared execution role used for job submission during the validation phase. Deploy it as follows:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The Scala sample lives at sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus. The CloudFormation template lives at resources/cloudformation/.

Deploy the CloudFormation template to create the target Amazon EMR Serverless application and a shared execution role:

aws cloudformation deploy \
  --template-file resources/cloudformation/emr-serverless-target-setup.yaml \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
  StagingBucketName=${STAGING_BUCKET_PATH} \
  TargetReleaseLabel=emr-spark-8.0.0 \
  TargetApplicationName=spark-upgrade-target

This creates an Amazon EMR Serverless target application (Spark 4.0) for upgrade validation, with a shared execution role. The application auto-stops after 15 minutes of idle time, so there is no cost when not in use. To upgrade between different Spark versions, override the SourceReleaseLabel and TargetReleaseLabel parameters with the Amazon EMR release labels that you want.

After the stack completes, note the outputs:

aws cloudformation describe-stacks \
  --stack-name spark-emr-serverless-upgrade \
  --region ${SMUS_MCP_REGION} \
  --query "Stacks[0].Outputs" --output table

This gives you the TargetApplicationId and ExecutionRoleArn needed for the upgrade prompt. Make a note of them.

2. Upgrade

This section covers a complete end-to-end upgrade using a representative ecommerce pipeline, a Scala application that processes order events, applies transformations, and writes results using merge-style upsert patterns. The same workflow applies to Scala and Spark SQL workloads covered in subsequent sections.

Step 1: Clone the sample project

Start by cloning the sample project from the AWS samples repository:

git clone https://github.com/aws-samples/sample-amazon-emr-spark4-examples
cd sample-amazon-emr-spark4-examples/scala3/demo_1_spark_change_focus

The repository includes representative PySpark, Scala, and Spark SQL applications designed to demonstrate common Spark 3.x patterns and their Spark 4.0 equivalents.

Step 2: Open in your IDE and connect to the MCP server

Open the project in Kiro or VS Code with the Cline extension. Verify that the dataprocessing-mcp server is active and connected. You can see it listed as an available MCP server in your IDE’s MCP panel. If you haven’t completed the one-time setup, follow the setup guide before proceeding.

Step 3: Start the upgrade with a natural language prompt

Once connected, initiate the upgrade by entering the following request in the agent interface:

Use the dataprocessing-mcp server to upgrade my local project at <path-to-your-project>.
Upgrade my Spark application from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/

The agent responds by invoking generate_spark_upgrade_plan, analyzing your project structure, identifying incompatible patterns, and presenting a prioritized upgrade plan before proceeding with any code changes.

After you confirm the plan, the agent proceeds autonomously through the remaining phases:

1. Build configuration update — update_build_configuration rewrites pom.xml, build.sbt, or requirements.txt to target Spark 4.0 dependencies.
2. Environment validation — Java and Python environments are checked and updated as needed.
3. Code transformation — fix_upgrade_failure applies targeted fixes for each identified incompatibility, iterating until the project compiles cleanly.
4. Remote validation — run_validation_job submits the upgraded application to your Amazon EMR Serverless target application and monitors execution through check_job_status.
5. Data quality check (optional) — get_data_quality_summary compares output between the Spark 3.5 baseline and the Spark 4.0 run, confirming correctness before sign-off.

With the sample Scala ecommerce pipeline cloned from the sample-amazon-emr-spark4-examples repository and your IDE connected to the MCP server, you submitted a natural language prompt. This triggered the agent to analyze the project structure and generate a prioritized five-step upgrade plan, all before making any code changes.

Now that the upgrade plan is in place, the following sections walk through the specific code transformations the agent applies for a Scala workload.

Scala workload migration

This section covers the complete migration of a representative Scala Spark application from Amazon EMR Serverless 6.9.0 (Spark 3.3, Scala 2.12) to Amazon EMR Serverless 8.0.0 (Spark 4.0, Scala 2.13), using the demo_1_spark_change_focus sample from the AWSSpark4AutoUpgradeDemo repository.

Sample project: Ecommerce product change focus pipeline

The sample application processes product catalog change events from Amazon S3, applies enrichment transformations, and writes aggregated results back to Amazon S3. It represents a common pattern in ecommerce data platforms: incremental processing of catalog updates with downstream aggregation. In this example, we use VS Code with Cline, but you can also use Kiro or any other MCP-enabled IDE.

Project structure:

demo_1_spark_change_focus/
├── build.sbt
├── project/
│   ├── build.properties
│   └── plugins.sbt
└── src/
    └── main/
        └── scala/
            └── job_script.scala

The following figure shows the project structure as it appears in the IDE, with the build configuration and Scala source files.

Project structure with build.sbt and Scala source files in the IDE

Step 1: Invoke the upgrade agent

Enter the following prompt:

Use the dataprocessing-mcp server to upgrade my local project at
/Users/<your-path>/AWSSpark4AutoUpgradeDemo/resources/scala/demo_1_spark_change_focus/
from Amazon EMR Serverless version 6.9.0 to Amazon EMR Serverless version 8.0.0.
Use Amazon EMR Serverless app-id <your-app-id> for validation.
Store artifacts at s3://amzn-s3-demo-bucket/spark4-upgrade/scala/

The agent reads your project files, identifies the current Spark and Scala versions, and generates a structured upgrade plan (the analysis ID is logged for traceability). The following figure shows the agent generating the upgrade plan.

Agent generating the upgrade plan with Spark and Scala version analysis

When prompted, choose Yes, proceed with the upgrade as planned, then choose Approve on the subsequent permission dialog to allow the agent to update the build configuration.

Step 2: Build configuration update — build.sbt

The first action of the agent is to invoke update_build_configuration, rewriting build.sbt to target Spark 4.0 and Scala 2.13, as shown in the following figure.

Agent output showing build.sbt updates from Spark 3.3 to Spark 4.0

The build configuration update is generated. The following are the recommended changes for build.sbt.

Changes summary:

1. Scala version: Updated from 2.12.15 to 2.13.16 (required for Spark 4.0).
2. Spark dependencies: Updated from 3.3 to 4.0.
   • spark-core: 3.3 → 4.0.
   • spark-sql: 3.3 → 4.0.
3. Assembly settings: Added configuration for creating uber JARs with proper merge strategies.
4. Dependency exclusions: Added rules to exclude provided dependencies (Spark, Scala, Hadoop) from assembly.

The following figure shows the updated build.sbt and plugins.sbt files after the configuration changes are saved.

Updated build.sbt and plugins.sbt files saved after configuration changes

Step 3: Java environment check

Amazon EMR Serverless 8.0.0 runs on Java 17. The agent invokes check_and_update_build_environment to verify your local Java Development Kit (JDK) and upgrade it from Java 11 to Java 17, as shown in the following figure.

Agent verifying Java environment and recommending JDK 17 for Amazon EMR 8.0

Step 4: Scala source code transformations

After updating the build configuration, the agent compiles the project and applies fix_upgrade_failure iteratively to resolve Scala 2.13 and Spark 4.0 breaking changes. Scala 2.13 removed several deprecated collection methods that were available in 2.12. The compilation failed with errors related to the Scala 2.13 syntax change. The .to[Set] syntax needs to be updated to .to(Set) for Scala 2.13. The agent used the fix_upgrade_failure tool to resolve the compilation errors. The following are the key transformations applied to job_script.scala.

The following figure shows the agent applying Scala 2.13 source code transformations to resolve the compilation errors.

Agent applying Scala 2.13 source code transformations to resolve compilation errors

// Disable ANSI (American National Standards Institute) (SQL compliance mode)(ANSI) mode to handle overflow and malformed cast operations
spark.conf.set("spark.sql.ansi.enabled", "false")

df.createOrReplaceTempView("airports")
// With ANSI mode disabled, overflow values will be handled gracefully
var new_df = spark.sql("SELECT *, CAST(build_time AS SMALLINT) as numeric_build_time FROM airports")
new_df.show()
new_df.createOrReplaceTempView("airports")

// Migration change: The to[Collection] method was replaced by the to(Collection) method.
val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)
println(airports_in_us)
val airports_in_us_java: java.util.Set[String] = airports_in_us.asJava

// With ANSI mode disabled, malformed CAST operations will return null instead of failing
new_df = spark.sql("SELECT *, CAST(code AS INT) as numeric_code FROM airports")
new_df.show()
new_df.write
  .mode("overwrite")
  .parquet(outputPath)

Before

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to[Set]

After

val airports_in_us: Set[String] = spark.sql("SELECT name FROM airports WHERE country='USA'").collect().map(_.getString(0)).to(Set)

Code change explanation:

• Scala 2.13 changed the collection conversion API. The .to[Collection] syntax was replaced with .to(Collection) using parentheses instead of square brackets.
• Updated collection conversion from .to[Set] to .to(Set) to comply with Scala 2.13+ syntax requirements.
• Changed import scala.collection.JavaConverters._ to import scala.jdk.CollectionConverters._ and updated .to[Set] to .to(Set).
• Renamed the object from Spark3_3_Job to Spark4_0_Job. Updated the Parquet config keys from spark.sql.legacy.parquet.int96RebaseModeInRead/Write to spark.sql.parquet.int96RebaseModeInRead/Write.
• Added spark.conf.set("spark.sql.ansi.enabled", "false") to handle overflow and malformed cast operations gracefully.
• The output path was updated to s3://xxxxxxxxx/output.

After the compilation succeeds, the agent builds the assembly JAR. Choose Save to create a report for the build result.

Step 5: Runtime validation

Provide the following information to run the validation job on Amazon EMR Serverless.

Amazon EMR Serverless application ID (target application running Spark 4.0 on Amazon EMR 8.0.0):

• To create an Amazon EMR application, follow the Amazon EMR documentation, or provide a prompt for the agent to create one for you.
• Format: 00xxxxxxxxxxxxxxxxxxxxxxxxxx.

Execution role Amazon Resource Name (ARN) (IAM role for the job):

• Set up the execution role following the IAM role guide.
• Format: arn:aws:iam::123456789012:role/YourRoleName.

Amazon S3 staging path (for uploading the JAR and storing results):

• Format: s3://amzn-s3-demo-bucket/path/.

AWS profile (the AWS profile to use for CLI commands, found in your mcp_settings.json file):

• Example: default, dev, pro.

After you submit this information, the agent uploads the JAR to Amazon S3 and submits the validation job with the following arguments:

{
  "analysis_id": "a8869720-e005-41b1-89f3-620e1c5663c0",
  "application_type": "EMR-Serverless",
  "compute_id": "xxxxxxxxxxxxxx",
  "compute_run_config": {
    "executionRoleArn": "arn:aws:iam::xxxxxxxx:role/data-processing-mcp-role",
    "jobDriver": {
      "sparkSubmit": {
        "entryPoint": "s3://xxxxx/xxxxxxx/xxxxxxxxx-xxxxxxx-xxxxxxxx/xxx-job-assembly-1.0.jar",
        "entryPointArguments": [],
        "sparkSubmitParameters": "--class Spark4_0_Job --conf spark.executor.cores=4 --conf spark.executor.memory=16g --conf spark.driver.cores=4 --conf spark.driver.memory=16g --conf spark.executor.instances=2"
      }
    },
    "configurationOverrides": {
      "monitoringConfiguration": {
        "cloudWatchLoggingConfiguration": {
          "enabled": true,
          "logGroupName": "/aws/emr-serverless"
        }
      }
    }
  },
  "enable_data_quality_check": false,
  "s3_staging_path": "s3://xxxxxxx/xxxxxxxxx/",
  "is_source_version_run": false,
  "spark_metadata_file_put_presigned_url": null,
  "static_artifacts_file_paths": null
}

The agent monitors the job status upon approval.

Amazon EMR Serverless job validation output showing successful completion

Once you receive a success response, the agent proceeds to generate the upgrade summary for your Spark project.

{"success":true,"message":"EMR SERVERLESS job completed successfully","compute_run_id":"xxxxxxxxxx","compute_id":"xxxxxxxxxx","status":"SUCCESS","application_type":"EMR-Serverless","next_action":"Proceed to next step in the plan"}

After the job succeeds, you can optionally cross-check the results by navigating to your Amazon S3 bucket to verify the output files were written successfully, or by reviewing the job run details in the Amazon EMR Serverless console to confirm execution status and logs. Choose Approve to allow the agent to save the summary to a local file and upload it to Amazon S3.

You can also request ANSI mode enablement through the agent interface, and it applies the necessary updates accordingly.

Relevant Spark changes:

• Migration rule: Spark 4.0 enables ANSI mode by default. To handle type conversion errors gracefully while keeping ANSI mode enabled, use TRY_CAST instead of CAST.
• Change description: Enabled ANSI mode and replaced CAST with TRY_CAST for operations that might fail, specifically timestamp-to-smallint overflow and string-to-int malformed value conversions.

Applied changes:

• Code diff — src/main/scala/job_script.scala:
  • Changed spark.conf.set("spark.sql.ansi.enabled", "false") to spark.conf.set("spark.sql.ansi.enabled", "true").
  • Replaced CAST(build_time AS SMALLINT) with TRY_CAST(build_time AS SMALLINT).
  • Replaced CAST(code AS INT) with TRY_CAST(code AS INT).

The agent compiles the change and follows the previous steps to run the job on Amazon EMR Serverless.

Result: SUCCESS

In this Scala workload migration section, the agent automatically upgraded the ecommerce pipeline from Spark 3.3 and Scala 2.12 on Amazon EMR 6.9.0 to Spark 4.0 and Scala 2.13 on Amazon EMR 8.0.0. It rewrote build.sbt and plugins.sbt, upgraded the JDK from 11 to 17, and applied Scala 2.13 syntax fixes (.to(Set), CollectionConverters), Parquet config key updates, and ANSI mode handling with TRY_CAST replacements. The upgraded JAR was compiled, submitted to Amazon EMR Serverless, and validated with a SUCCESS status, completing the full migration without manual code edits.

Clean up

To avoid ongoing charges, delete the resources created during this walkthrough. Start by emptying the Amazon S3 staging bucket, then delete both AWS CloudFormation stacks in reverse order:

1. Empty the Amazon S3 staging bucket.

   aws s3 rm s3://${STAGING_BUCKET_PATH} --recursive

2. Delete the Amazon EMR Serverless application stack.

   aws cloudformation delete-stack --stack-name spark-emr-serverless-upgrade

3. Delete the MCP setup stack (IAM role and Amazon S3 bucket).

   aws cloudformation delete-stack --stack-name spark-upgrade-mcp-setup

Conclusion

The AWS Spark Upgrade Agent transforms what has traditionally been a months-long, error-prone migration process into an automated, IDE-driven workflow that completes in hours. By combining intelligent code analysis, targeted transformations, and an iterative local-to-remote validation loop, the agent handles the complexity of upgrading Scala workloads from Spark 3.x to Spark 4.0 on Amazon EMR 8.x. The demo_1_spark_change_focus walkthrough demonstrates the ability of the agent to automatically update build configurations, apply Scala 2.13 syntax changes, handle Spark 4.0 breaking changes like ANSI mode defaults, and validate results against live Amazon EMR clusters, all through natural language prompts in your IDE. For teams managing large-scale Spark estates, this approach eliminates manual debugging cycles, reduces migration risk, and unlocks the performance gains of Spark 4.0 without the traditional engineering overhead.

Next steps:

• If you’re new to the Spark Upgrade Agent, start with the introduction post for a lighter-weight introduction before tackling Scala workloads.
• For a complete PySpark implementation and demo, refer to Upgrade PySpark from Spark 3.5 to Spark 4.0 with AWS Spark Upgrade Agent.
• When you are ready for production, review the security model in the Architecture section and the IAM role setup guide to confirm your least-privilege configuration before running against production workloads.

Useful resources:

Have questions or feedback? Share your migration experience in the AWS re:Post community or open an issue in the sample repository. We’d love to hear how the agent performs on your workloads.

About the authors