Spark Release 3.4.0

Apache Spark 3.4.0 is the fifth release of the 3.x line. With tremendous contribution from the open-source community, this release managed to resolve in excess of 2,600 Jira tickets.

This release introduces Python client for Spark Connect, augments Structured Streaming with async progress tracking and Python arbitrary stateful processing, increases Pandas API coverage and provides NumPy input support, simplifies the migration from traditional data warehouses by improving ANSI compliance and implementing dozens of new built-in functions, and boosts development productivity and debuggability with memory profiling.

To download Apache Spark 3.4.0, visit the downloads page. You can consult JIRA for the detailed changes. We have curated a list of high level changes here, grouped by major modules.

Highlight

  • Python client for Spark Connect (SPARK-39375)
  • Implement support for DEFAULT values for columns in tables (SPARK-38334)
  • Support TIMESTAMP WITHOUT TIMEZONE data type (SPARK-35662)
  • Support “Lateral Column Alias References” (SPARK-27561)
  • Harden SQLSTATE usage for error classes (SPARK-41994)
  • Enable Bloom filter Joins by default (SPARK-38841)
  • Better Spark UI scalability and Driver stability for large applications (SPARK-41053)
  • Async Progress Tracking in Structured Streaming (SPARK-39591)
  • Python Arbitrary Stateful Processing in Structured Streaming (SPARK-40434)
  • Pandas API coverage improvements (SPARK-42882) and NumPy input support in PySpark (SPARK-39405)
  • Provide a memory profiler for PySpark user-defined functions (SPARK-40281)
  • Implement PyTorch Distributor (SPARK-41589)
  • Publish SBOM artifacts (SPARK-41893)
  • Support IPv6-only environment (SPARK-39457)
  • Customized K8s Scheduler (Apache YuniKorn and Volcano) GA (SPARK-42802)

Spark SQL

Features

ANSI Compliance

  • ANSI SQL mode: always return null on invalid access to map column (SPARK-40066)
  • Support double quoted identifiers (SPARK-40585)
  • ANSI SQL mode: Round/Bround should return an error on integer overflow (SPARK-42045)
  • Support casting of integrals to ANSI intervals (SPARK-40008)
  • Support cast of decimals to ANSI intervals (SPARK-40014)
  • Return wider ANSI interval types from the percentile functions (SPARK-40151)
  • Support cast of ANSI intervals to decimals (SPARK-39470)
  • Support casting intervals to integrals in ANSI mode (SPARK-39451)
  • Harden SQLSTATE usage for error classes (SPARK-41994)

Functions

  • Support table-valued generator functions in the FROM clause (SPARK-41594)
  • Support ANSI Aggregate Function: REGR_SXY (SPARK-37681)
  • Support ANSI Aggregate Function: REGR_R2 (SPARK-37641)
  • Support ANSI Aggregate Function: REGR_SXX (SPARK-37672)
  • Support ANSI Aggregate Function: REGR_SYY (SPARK-37702)
  • Support ANSI Aggregate Function: REGR_SLOPE (SPARK-39230)
  • Support ANSI Aggregate Function: REGR_INTERCEPT (SPARK-37623)
  • Support ANSI aggregation function PERCENTILE_CONT as window function (SPARK-38219)
  • Support ANSI Aggregation Function: PERCENTILE_DISC (SPARK-37691)
  • Support SPLIT_PART function (SPARK-38063)
  • Support TRY_AVG function (SPARK-38589)
  • Support TRY_TO_BINARY function (SPARK-38590)
  • Support the TO_NUMBER and TRY_TO_NUMBER SQL functions according to a new specification (SPARK-38796)
  • Support ANSI general value specification and function - USER (SPARK-39138)
  • Support TO_CHAR and TRY_TO_CHAR functions to format Decimal values as strings (SPARK-28516)
  • Support ANY_VALUE aggregate function (SPARK-39213)
  • Support EQUAL_NUL function (SPARK-39305)
  • Support aggregate function MEDIAN (SPARK-39320)
  • Support REGEXP_COUNT function (SPARK-39618)
  • Support REGEXP_INSTR function (SPARK-39744)
  • Support REGEXP_SUBSTR function (SPARK-39695)
  • Support UNPIVOT function (SPARK-39876)
  • Support TRY_TO_TIMESTAMP function (SPARK-39795)
  • Support url encode/decode as built-in function and tidy up url-related functions (SPARK-39741)
  • Support aggregate function MODE (SPARK-39808)
  • Support GET function (SPARK-40109)
  • Add function aliases: LEN, DATEPART, DATEADD, DATE_DIFF, CURDATE (SPARK-40352)
  • Improve the TO_BINARY function (SPARK-40112)
  • Support CURRENT_SCHEMA (SPARK-41323)
  • Support data masking built-in function MASK (SPARK-40687)
  • Support high-order function: ARRAY_COMPACT (SPARK-41235)
  • Support ARRAY_APPEND function (SPARK-41232)
  • Support ARRAY_INSERT function (SPARK-41234)
  • Support LUHN_CHECK function (SPARK-42191)
  • Support ARRAY_SORT(column, comparator) (SPARK-39925)

Data Sources

  • Support Column Stats in DS v2 (SPARK-41378)
  • Storage Partitioned Join (SPJ) in DS v2 (SPARK-37375)
  • Row-level operations in DS v2 (SPARK-35801)
  • Add SupportsReportOrdering mix in interface for DS v2 Scan (SPARK-38647)
  • Infer DATE type for CSV schema inference (SPARK-39469)
  • Support driver metrics in DS v2 custom metric API (SPARK-39635)
  • Distribution and ordering support DS v2 function in writing (SPARK-39607)
  • StringEndsWith/Contains support push down to Parquet so that we can leverage dictionary filter (SPARK-39002)
  • Support UDT in Spark Parquet vectorized reader (SPARK-39086)
  • Extend METADATA column to support row indexes for Parquet files (SPARK-37980)
  • Support reading parquet FIXED_LEN_BYTE_ARRAY type (SPARK-41096)
  • Optimize the order of filtering predicates (SPARK-40045)
  • Support CTE and temp table queries with MSSQL JDBC (SPARK-37259)
  • Support ignoreCorruptFiles and ignoreMissingFiles in Data Source options (SPARK-38767)
  • Pull out v1 write to WriteFiles (SPARK-41407)
  • Add read-side char padding to cover external data files (SPARK-40697)

Query Optimization

  • Merge non-correlated scalar subqueries (SPARK-34079)
  • Enable Bloom filter Joins by default (SPARK-38841)
  • Remove unnecessary distinct in aggregate expression by distinctKeys (SPARK-38832)
  • Support predicate pushdown and column pruning for de-duped CTEs (SPARK-37670)
  • Remove outer join if aggregate functions are duplicate agnostic on streamed side (SPARK-38886)
  • Remove left/right outer join if only left/right side columns are selected and the join keys on the other side are unique (SPARK-39172)
  • Optimize global Sort to RepartitionByExpression (SPARK-39911)
  • Optimize TransposeWindow rule (SPARK-38034)
  • Enhance EliminateSorts to support removing sorts via LocalLimit (SPARK-40050)
  • Push local limit to both sides if join condition is empty (SPARK-40040)
  • Add PushProjectionThroughLimit for Optimizer (SPARK-40501)
  • Support PIVOT/UNPIVOT with join children (SPARK-41195)
  • Support column pruning with multiple nondeterministic Filters (SPARK-41017)
  • Improve output partitioning and ordering with AQE cache (SPARK-41048)
  • Improve multi like performance by creating a balanced expression tree predicate (SPARK-41167)
  • Remove the Sort if it is the child of RepartitionByExpression (SPARK-36703)
  • Use available column statistics from completed query stages (SPARK-39991)
  • Reuse expressions in WindowSpecDefinition (SPARK-41805)
  • Improve AliasAwareOutputPartitioning and AliasAwareQueryOutputOrdering to take all aliases into account (SPARK-40086, SPARK-42049)
  • Push down limit through Python UDFs (SPARK-42115)

Code Generation and Query Execution

  • Make defaultJoin in BroadcastNestedLoopJoinExec running in parallel (SPARK-40487)
  • Codegen Support for HiveGenericUDF (SPARK-42051)
  • Introduce shuffle on SinglePartition (SPARK-41986)
  • Makes DPP support the pruning side has Union (SPARK-39217)

Other Notable Changes

  • Support Auto Partition Statistics Collection (SPARK-38573)
  • Format error messages in the Thrift Server (SPARK-40098)
  • Add an extension API to do plan normalization for caching (SPARK-41183)
  • Refactor Spark types by introducing physical types (SPARK-41226)
  • OneOf field support and recursion checks (SPARK-41396)
  • Centralize the column resolution logic (SPARK-41405)
  • Improve the plan change validation (SPARK-42081)
  • Introduce SparkPath for typesafety (SPARK-41970)
  • Throw Exception for db_name.view_name when creating temp view by Dataset API (SPARK-41090)
  • Change the default value of argument of Mask function from -1 to NULL (SPARK-42070)

Spark Core

Decommission

  • Avoid unnecessary task rerun on decommissioned executor lost if shuffle data migrated (SPARK-41469)
  • Ignore stage fetch failure caused by decommissioned executor (SPARK-40481)
  • Enable spark.storage.decommission.(rdd shuffle)Blocks.enabled by default (SPARK-40198)
  • Add support for YARN decommissioning when ESS is disabled (SPARK-30835)

Scheduler

  • Make stage scheduling support local-cluster mode (SPARK-41949)
  • Support stage level task resource profile for standalone cluster when dynamic allocation disabled (SPARK-39853)
  • Delay onDisconnected to enable Driver receives ExecutorExitCode (SPARK-39957)
  • Improve the speculation through the stage task metrics (SPARK-32170)
  • Add stage level resource scheduling support for standalone cluster (SPARK-39062)
  • Improve LaunchTask process to avoid Stage failures caused by fail-to-send LaunchTask messages (SPARK-39955)

Shuffle

  • Add Push Based Shuffle client side read metrics (SPARK-36620)
  • Shuffle server side metrics for Push-based shuffle (​​SPARK-33573)
  • Ensure mergedShuffleCleaner have been shutdown before db close (SPARK-40186)
  • Add RocksDB support for shuffle service state store (SPARK-38888)
  • Encapsulate LevelDB used to store remote/external shuffle state as DB (SPARK-38909)
  • Enable spark.dynamicAllocation.shuffleTracking.enabled by default (SPARK-3984)
  • Enable Push-based shuffle service to store state in NM level DB for work preserving restart (SPARK-33236)
  • Remove shuffle blocks using the shuffle service for released executors (SPARK-37618)

Other Notable Changes

  • Support IPv6-only environment (SPARK-39457)
  • Enable spark.kryo.unsafe by default (SPARK-42137)
  • Disallow arbitrary custom classpath with proxy user in cluster mode (SPARK-41958)
  • Avoid BlockManager re-registration if the executor has been lost (SPARK-41360)
  • Remove the limitation that single task result must fit in 2GB (SPARK-40622)
  • Remove the support of deprecated spark.akka.* configs (SPARK-40401)
  • Change default logging to stderr to consistent with the behavior of log4j (SPARK-40406)
  • Exclude DirectTaskResult metadata when calculating result size (SPARK-40261)
  • Allow customize initial partitions number in take() behavior (SPARK-40211)
  • Use interruptible lock instead of synchronized in Executor.updateDependencies() (SPARK-40235)
  • Task failure should always trigger task failure listeners (SPARK-40106)
  • Add the ability to selectively disable watching or polling (SPARK-36462)
  • Do not cache unserialized broadcast relations on the driver (SPARK-39983)
  • Fix deadlock between TaskMemoryManager and UnsafeExternalSorter.SpillableIterator (SPARK-39283)
  • Expose the number partitions in a stage to TaskContext (SPARK-38679)
  • Make memory overhead factor configurable (SPARK-38194)
  • Avoid using bash -c in ShellBasedGroupsMappingProvider (SPARK-38992)

Structured Streaming

Major Features

  • Async Progress Tracking (SPARK-39591)
  • Python Arbitrary Stateful Processing in Structured Streaming (SPARK-40434)
  • Protobuf Support in Structured Streaming (SPARK-40653)
  • Fix late record filtering to support chaining of stateful operators (SPARK-40925)

Other Notable Changes

  • Introducing a streaming checkpoint file manager based on Hadoop’s Abortable interface (SPARK-40039)
  • Deprecate Trigger.Once and Promote Trigger.AvailableNow (SPARK-39805)
  • Expose the information of catalog table to the logical plan in streaming query (SPARK-39564)
  • Support collecting metrics from streaming sinks (SPARK-38564)
  • Deprecate DStream API (SPARK-42075)
  • Flip the default value of Kafka offset fetching config (SPARK-40844)
  • Provide cloned spark session in DataFrame in user function for foreachBatch sink in PySpark (SPARK-41379)

Spark Connect

Python Client

Scala Client

PySpark

Pandas API on Spark

Other Notable Changes

MLLIB

SparkR

Live UI and History Server Service

  • Better Spark UI scalability and Driver stability for large applications(SPARK-41053)
  • Use RocksDB for spark.history.store.hybridStore.diskBackend by default (SPARK-42277)
  • Group nested executions under the root execution(SPARK-41752)
  • Show metrics properties in the environment tab(SPARK-39110)
  • Fix StagePage input size/records not show when records greater than zero(SPARK-34777)
  • Improve event logging JsonProtocol performance by using Jackson instead of Json4s (SPARK-39489)
  • Support spark.history.fs.update.batchSize (SPARK-39225)

Build

Credits

Last but not least, this release would not have been possible without the following contributors: Abhishek Dixit, Abu Bakr Siddiq, Adam Binford, Ahmed Mahran, Aimilios Tsouvelekakis, Ait Zeouay Amrane, Aki Sukegawa, Ala Luszczak, Alex Balikov, Alkis Evlogimenos, Allan Folting, Allison Portis, Allison Wang, Andrew Ray, Andy Grove, Andy Lam, Anish Shrigondekar, Ankit Prakash Gupta, Anton Ippolitov, Anton Okolnychyi, Aravind Patnam, Artsiom Yudovin, Arvin Zheng, Attila Zsolt Piros, Austin Wang, Ben Zhang, Bjorn Jorgensen, Bjørn Jørgensen, Bo Zhang, Bobby Wang, Brandon Dahler, Brennan Stein, Brian Schaefer, Brian Yue, Bruce Robbins, Carmen Kwan, Chandni Singh, Chao Sun, Chaoqin Li, Cheng Pan, Cheng Su, Chenhao Li, Chris Nauroth, Daniel Davies, Daniel Fiterman, Daniel Ranchal Parrado, Daniel Tenedorio, David Lewis, Dch Nguyen, Deepyaman Datta, Dennis Huo, Deshan Xiao, Desmond Cheong, Dongjoon Hyun, Dustin William Smith, ELHoussineT, Emil Ejbyfeldt, Enrico Minack, Erik Krogen, Eugene-Mark, Frank Yin, Fred Liu, Fredrik Mile, Fu Chen, Furcy Pin, Gabor Roczei, Gautham Banasandra, Gengliang Wang, Gidon Gershinsky, Guangxin Wang, Haejoon Lee, Hai Tao, Herman Van Hovell, Hisoka-X, Holden Karau, Huanli Wang, Hui An, Hyukjin Kwon, Immanuel Buder, Ismaël Mejía, Ivan Sadikov, Jack Chen, Jatin Sharma, Jeffrey Chen, Jelmer Kuperus, Jerry Peng, Jiaan Geng, JiexingLi, Johan Lasperas, John Caveman, John Zhuge, Jonathan Cui, Josh Rosen, Jove Yuan, Juliusz Sompolski, Jungtaek Lim, Kai-Hsun Chen, Kapil Kumar Singh, Karen Feng, Karuppayya Rajendran, Kazuaki Ishizaki, Kazuyuki Tanimura, Kelvin Jiang, Kent Yao, Keunhyun Oh, Khaled Hammouda, Khalid Mammadov, Kian Eliasi, Kimahriman, Kris Mok, Kumar, Pralabh, Kun Wan, Lee Yang, Liang-Chi Hsieh, Lingyun Yuan, Linhong Liu, Lorenzo Martini, LorenzoMartini, Luca Canali, Maciej Szymkiewicz, Manu Zhang, Mark Khaitman, Martin Grund, Martin Tzvetanov Grigorov, Maryann Xue, Max Gekk, Maya Anderson, Minchu Yang, Mridul Muralidharan, NarekDW, Nicholas Chammas, Niranjan Jayakar, Ole Sasse, Oleksiy Dyagilev, PJ Fanning, Peter Toth, Prashant Singh, Raghu Angadi, Rakesh Raushan, Reynold Xin, Rithwik Ediga Lakhamsani, Robert (Bobby) Evans, Rui Wang, Ruifeng Zheng, Runyao Chen, Ryan Johnson, Sandeep Katta, Sandeep Singh, SandishKumarHN, Santosh Pingale, Sean Owen, Serge Rielau, Shaoyun Chen, Shardul Mahadik, Shiqi Sun, Shixiong Zhu, Shrikant Prasad, Shuyou Dong, Stefaan Lippens, Steve Loughran, Steven Aerts, Sumeet Gajjar, Supun Nakandala, Swaminathan Balachandran, Takuya Ueshin, Tanel Kiis, Ted Yu, Tengfei Huang, Thejdeep Gudivada, Tobias Stadler, Tom Van Bussel, Tristan Nixon, Venki Korukanti, Vinod KC, Vitalii Li, Vivek Atal, Warren Zhu, Wei Liu, Weichen Xu, Weiwei Yang, Wenchen Fan, Wenli Looi, Wilfred Spiegelenburg, William Hyun, William Zijie, WolverineJiang, Xiduo You, Xing Lin, Xingbo Jiang, Xingchao, Zhang, Xinrong Meng, Xinyi Yu, XiuLi Wei, Yan Wei, Yang Jie, Yaohua628, Ye Zhou, Yi Wu, Yi Zhu, Yikf, Yikun Jiang, Yubi Lee, Yuming Wang, Zach Schuermann, Zhen Li, Zhen Wang, Zhiming She, Ziqi Liu, awdavidson, beobest2, bjornjorgensen, bzhaoopenstack, carlfu-db, cashmand, chenzhx, constzhou, dcoliversun, dengziming, fanyilun, fred-db, ganeshchand, gaoyajun02, guanziyue, harupy, huangxiaopingRD, huaxingao, idealspark, jackylee-ch, jiang13021, jiaoqingbo, khalidmammadov, kuwii, leesf, lvshaokang, lw33, mattshma, mcdull-zhang, minyyy, moritzkoerber, morvenhuang, mridulm, nyingping, panbingkun, philwalk, pralabhkumar, qiuliang988, santosh-d3vpl3x, seunggabi, smallzhongfeng, sus, thyecust, toujours33, uchiiii, utkarsh39, vicennial, wangshengjie123, wankunde, wayneguow, wecharyu, weiyuyilia, wineternity, wzx140, xiaonanyang-db, xiuzhu9527, yabola, yikf, zekai-li, zhangbutao, zheniantoushipashi, zhixingheyi-tian, zhouyifan279, zwangsheng, zzzzming95


Spark News Archive

Latest News

Archive