Optimize Amazon EMR runtime for Apache Spark with EMR S3A

With the Amazon EMR 7.10 runtime, Amazon EMR has launched EMR S3A, an improved implementation of the open supply S3A file system connector. This enhanced connector is now routinely set because the default S3 file system connector for Amazon EMR deployment choices, together with Amazon EMR on EC2, Amazon EMR Serverless, Amazon EMR on Amazon EKS, and Amazon EMR on AWS Outposts, sustaining full API compatibility with open supply Apache Spark.

Within the Amazon EMR 7.10 runtime for Apache Spark, the EMR S3A connector reveals efficiency similar to EMRFS for learn workloads, as demonstrated by TPC-DS question benchmark. The connector’s most vital efficiency beneficial properties are evident in write operations, with a 7% enchancment in static partition overwrites and a 215% enchancment for dynamic partition overwrites when in comparison with EMRFS. On this publish, we showcase the improved learn and write efficiency benefits of utilizing Amazon EMR 7.10.0 runtime for Apache Spark with EMR S3A as in comparison with EMRFS and the open supply S3A file system connector.

Learn workload efficiency comparability

To guage the learn efficiency, we used a take a look at surroundings based mostly on Amazon EMR runtime model 7.10.0 working Spark 3.5.5 and Hadoop 3.4.1. Our testing infrastructure featured an Amazon Elastic Compute Cloud (Amazon EC2) cluster comprised of 9 r5d.4xlarge situations. The first node has 16 vCPU and 128 GB reminiscence, and the eight core nodes have a complete of 128 vCPU and 1024 GB reminiscence.

The efficiency analysis was carried out utilizing a complete testing methodology designed to offer correct and significant outcomes. For the supply knowledge, we selected the three TB scale issue, which incorporates 17.7 billion information, roughly 924 GB of compressed knowledge partitioned in Parquet file format. The setup directions and technical particulars could be discovered within the GitHub repository. We used Spark’s in-memory knowledge catalog to retailer metadata for TPC-DS databases and tables.

To provide a good and correct comparability between EMR S3A vs. EMRFS and open supply S3A implementations, we carried out a three-phase testing method:

• Section 1: Baseline efficiency:
  • Established a baseline utilizing default Amazon EMR configuration with EMR’s S3A connector
  • Created a reference level for subsequent comparisons
• Section 2: EMRFS evaluation:
  • Maintained the default file system as EMRFS
  • Preserved different configuration settings
• Section 3: Open supply S3A testing:
  • Modified solely the hadoop-aws.jar file by changing it with the open supply Hadoop S3A 3.4.1 model
  • Maintained similar configurations throughout different elements

This managed testing surroundings was essential for our analysis for the next causes:

• We might isolate the efficiency impression particularly to the S3A connector implementation
• It eliminated potential variables that might skew the outcomes
• It offered correct measurements of efficiency enhancements between Amazon’s S3A implementation and the open supply different

Take a look at execution and outcomes

All through the testing course of, we maintained consistency in take a look at circumstances and configurations, ensuring any noticed efficiency variations could possibly be immediately attributed to the S3A connector implementation variations. A complete of 104 SparkSQL queries had been run in 10 iterations sequentially, and a mean of every question’s runtime in these 10 iterations was used for comparability. The typical of the ten iterations’ runtime on the Amazon EMR 7.10 runtime for Apache Spark with EMR S3A was 1116.87 seconds, which is 1.08 instances sooner than open supply S3A and comparable with EMRFS. The next determine illustrates the whole runtime in seconds.

The next desk summarizes the metrics.

*Detailed value estimates are mentioned later on this publish.

The next chart demonstrates the per-query efficiency enchancment of EMR S3A relative to open supply S3A on the Amazon EMR 7.10 runtime for Apache Spark. The extent of the speedup varies from one question to a different, with the quickest as much as 1.51 instances sooner for q3, with Amazon EMR S3A outperforming open supply S3A. The horizontal axis arranges the TPC-DS 3TB benchmark queries in descending order based mostly on the efficiency enchancment seen with Amazon EMR, and the vertical axis depicts the magnitude of this speedup as a ratio.

Learn value comparability

Our benchmark outputs the whole runtime and geometric imply figures to measure the Spark runtime efficiency. The fee metric can present us with further insights. Value estimates are computed utilizing the next formulation. They consider Amazon EC2, Amazon Elastic Block Retailer (Amazon EBS), and Amazon EMR prices, however don’t embrace Amazon Easy Storage Service (Amazon S3) GET and PUT prices.

• Amazon EC2 value (embrace SSD value) = variety of situations * r5d.4xlarge hourly charge * job runtime in hours
  • r5d.4xlarge hourly charge = $1.152 per hour
• Root Amazon EBS value = variety of situations * Amazon EBS per GB-hourly charge * root EBS quantity measurement * job runtime in hours
• Amazon EMR value = variety of situations * r5d.4xlarge Amazon EMR value * job runtime in hours
  • r5d.4xlarge Amazon EMR value = $0.27 per hour
• Complete value = Amazon EC2 value + root Amazon EBS value + Amazon EMR value

The next desk summarizes these prices.

Write workload efficiency comparability

We carried out benchmark exams to evaluate the write efficiency of the Amazon EMR 7.10 runtime for Apache Spark.

Static desk/partition overwrite

We evaluated the static desk/partition overwrite write efficiency of the completely different file system by executing the next INSERT OVERWRITE Spark SQL question. The SELECT * FROM vary(...) clause generated knowledge at execution time. This produced roughly 15 GB of knowledge throughout precisely 100 Parquet information in Amazon S3.

SET rows=4e9; -- 4 Billion
SET partitions=100;
INSERT OVERWRITE DIRECTORY 's3://${bucket}/perf-test/${trial_id}'
USING PARQUET SELECT * FROM vary(0, ${rows}, 1, ${partitions});

The take a look at surroundings was configured as follows:

• EMR cluster with emr-7.10.0 launch label
• Single m5d.2xlarge occasion (main group)
• Eight m5d.2xlarge situations (core group)
• S3 bucket in the identical AWS Area because the EMR cluster
• The trial_id property used a UUID generator to keep away from battle between take a look at runs

Outcomes

After working 10 trials for every file system, we captured and summarized question runtimes within the following chart. Whereas EMR S3A averaged solely 26.4 seconds, the EMRFS and open supply S3A averaged 28.4 seconds and 31.4 seconds—a 1.07 instances and 1.19 instances enchancment, respectively.

Dynamic partition overwrite

We additionally evaluated the write efficiency by executing the next INSERT OVERWRITE dynamic partition Spark SQL question, which joins TPC-DS 3TB partitioned Parquet knowledge of the desk web_sales and date_dim tables, which inserts roughly 2,100 partitions, the place every partition incorporates one Parquet file with a mixed measurement of roughly 31.2 GB in Amazon S3.

SET spark.sql.sources.partitionOverwriteMode=DYNAMIC;
INSERT OVERWRITE TABLE  PARTITION(wsdt_year,wsdt_month, wsdt_day) 
SELECT ws_order_number,ws_quantity,ws_list_price,ws_sales_price,
ws_net_paid_inc_ship_tax,ws_net_profit,dt.d_year as wsdt_year,dt.d_moy 
as wsdt_month,dt.d_dom as wsdt_day FROM web_sales, date_dim dt 
WHERE ws_sold_date_sk = d_date_sk;

The take a look at surroundings was configured as follows:

• EMR cluster with emr-7.10.0 launch label
• Single r5d.4xlarge occasion (grasp group)
• 5 r5d.4xlarge situations (core group)
• Roughly 2,100 partitions with one Parquet file every
• Mixed measurement of roughly 31.2 GB in Amazon S3

Outcomes

After working 10 trials for every file system, we captured and summarized question runtimes within the following chart. Whereas EMR S3A averaged solely 90.9 seconds, the EMRFS and open supply S3A averaged 286.4 seconds and 1,438.5 seconds—a 3.15 instances and 15.82 instances enchancment, respectively.

Abstract

Amazon EMR persistently enhances its Apache Spark runtime and S3A connector, delivering steady efficiency enhancements that assist large knowledge clients execute analytics workloads extra cost-effectively. Past efficiency beneficial properties, the strategic shift to S3A introduces important benefits, together with enhanced standardization, improved cross-platform portability, and strong community-driven help—all whereas sustaining or surpassing the efficiency benchmarks established by the earlier EMRFS implementation.

We advocate that you just keep up-to-date with the newest Amazon EMR launch to benefit from the newest efficiency and have advantages. Subscribe to the AWS Massive Knowledge Weblog’s RSS feed to be taught extra in regards to the Amazon EMR runtime for Apache Spark, configuration finest practices, and tuning recommendation.

Concerning the authors