pyspark.sql.functions.exists

pyspark.sql.functions.exists(col: ColumnOrName, f: Callable[[pyspark.sql.column.Column], pyspark.sql.column.Column]) → pyspark.sql.column.Column

Returns whether a predicate holds for one or more elements in the array.

New in version 3.1.0.

Changed in version 3.4.0: Supports Spark Connect.

Parameters

colColumn or str

name of column or expression

ffunction

(x: Column) -> Column: ... returning the Boolean expression. Can use methods of Column, functions defined in pyspark.sql.functions and Scala UserDefinedFunctions. Python UserDefinedFunctions are not supported (SPARK-27052).

Returns

Column

True if “any” element of an array evaluates to True when passed as an argument to given function and False otherwise.

Examples

>>> df = spark.createDataFrame([(1, [1, 2, 3, 4]), (2, [3, -1, 0])],("key", "values"))
>>> df.select(exists("values", lambda x: x < 0).alias("any_negative")).show()
+------------+
|any_negative|
+------------+
|       false|
|        true|
+------------+

previous

pyspark.sql.functions.transform

next

pyspark.sql.functions.forall