Spark Release 1.3.1

Spark 1.3.1 is a maintenance release containing stability fixes. This release is based on the branch-1.3 maintenance branch of Spark. We recommend all 1.3.0 users to upgrade to this stable release. Contributions to this release came from 60 developers.

To download Spark 1.3.1 visit the downloads page.

Fixes

Spark 1.3.1 contains several bug fixes in Spark SQL and assorted fixes in other components. Some of the more important fixes are highlighted below. You can visit the Spark issue tracker for the full list of fixes.

Spark SQL

  • Unable to use reserved words in DDL (SPARK-6250)
  • Parquet no longer caches metadata (SPARK-6575)
  • Bug when joining two Parquet tables (SPARK-6851)
  • Unable to read parquet data generated by Spark 1.1.1 (SPARK-6315)
  • Parquet data source may use wrong Hadoop FileSystem (SPARK-6330)

Spark Streaming

  • Potential for data loss during WAL recovery (SPARK-6222)

PySpark

  • Potential program hang when calling collect (SPARK-6667)

Spark Core

  • Thread safety issue in Netty shuffle (SPARK-6578)
  • Memory leak in output committer map (SPARK-6737)
  • Unable to perform fetch file when local directories run NFS (SPARK-6313)
  • NPE when cancelling and using mix of job groups (SPARK-6414)

Contributors

The following developers contributed to this release:

  • Adam Budde – Bug fixes in SQL
  • Andrew Or – Bug fixes in Core
  • Andrey Zagrebin – Improvement in SQL
  • Bill Chambers – Documentation in Core
  • Cheng Lian – Bug fixes and improvement in SQL
  • Chet Mancini – Improvements in Core and SQL
  • Christophe Preaud – Documentation in Core and YARN
  • Daoyuan Wang – New features in SQL
  • Davies Liu – Improvements in PySpark and SQL; bug fixes in tests, PySpark, and SQL; improvement in SQL
  • Dean Chen – Bug fixes in Core
  • Doing Done – Bug fixes in Core and SQL
  • Hung Lin – Bug fixes in scheduler
  • Ilya Ganelin – Improvements in Core
  • Imran Rashid – Bug fixes in Core
  • Iulian Dragos – Bug fixes in Core
  • Jayson Sunshine – Documentation in Core
  • Jeremy Freeman – Bug fixes in Streaming and MLlib
  • Jongyoul Lee – Improvements in Mesos; bug fixes in Core
  • Joseph K. Bradley – Documentation in PySpark, Streaming, SQL, MLlib, and Core
  • Josh Rosen – Improvements in Core; bug fixes in Java API, Core, scheduler, and Streaming
  • Kai Sasaki – Documentation in Core and MLlib; bug fixes in MLlib and PySpark
  • Kalle Jepsen – Improvements in PySpark
  • Kamil Smuga – Bug fixes in Core and PySpark
  • Kay Ousterhout – Bug fixes in Core, tests, and Web UI
  • Kevin (Sangwoo) Kim – Bug fixes in Core
  • Kousuke Saruta – Improvements in Streaming and tests
  • Lev Khomich – Improvements in Core
  • Liang-Chi Hsieh – Bug fixes in SQL
  • Liangliang Gu – Bug fixes in spark submit
  • Lomig Megard – Documentation in Core
  • Marcelo Vanzin – Bug fixes in Core and YARN
  • Matt Aasted – Bug fixes in EC2
  • Michael Armbrust – Improvements in SQL; documentation in Core; bug fixes in SQL; improvement in Core and SQL
  • Michael Griffiths – Bug fixes in Windows and Core
  • Milan Straka – Bug fixes in PySpark
  • Nan Zhu – Bug fixes in Core and SQL
  • Nathan McCarthy – Bug fixes in Core
  • Pei-Lun Lee – Bug fixes in SQL
  • Peter Parente – Improvements in Core
  • Peter Rudenko – Documentation in Core
  • Reynold Xin – Improvements in Core; documentation in Core; bug fixes in Core; improvement in SQL
  • Sean Owen – Bug fixes in Core, tests, and SQL
  • Shixiong Zhu – Bug fixes in Core
  • Tathagata Das – Improvements in Core and Streaming; bug fixes in Streaming
  • Thomas Graves – Bug fixes in Core
  • Tijo Thomas – Bug fixes in Core and SQL
  • Venkata Ramana Gollamudi – Bug fixes in SQL
  • Vinod KC – Bug fixes in Core and SQL
  • Volodymyr Lyubinets – Improvements and bug fixes in SQL
  • Xiangrui Meng – New features in MLlib and PySpark; bug fixes in PySpark, MLlib, and SQL; documentation in Core and MLlib
  • Yadong Qi – Improvements in SQL
  • Yanbo Liang – Bug fixes in MLlib and SQL
  • Yash Datta – Improvements in SQL
  • Yin Huai – Improvements and bug fixes in SQL
  • Yp Cat – Bug fixes in SQL
  • Yu ISHIKAWA – Improvements in MLlib
  • Yuri Saito – Bug fixes in SQL
  • Zhang, Liye – Bug fixes in Core and Web UI
  • Zhichao Li – Bug fixes in Streaming and Web UI
  • Zhichao Zhang – Documentation in Core

Thanks to everyone who contributed!


Spark News Archive

Latest News

Archive