Package org.apache.spark.sql.types
Class DataType
Object
org.apache.spark.sql.types.DataType
- Direct Known Subclasses:
ArrayType
,BinaryType
,BooleanType
,CalendarIntervalType
,CharType
,DateType
,DayTimeIntervalType
,MapType
,NullType
,NumericType
,ObjectType
,StringType
,StructType
,TimestampNTZType
,TimestampType
,UserDefinedType
,VarcharType
,YearMonthIntervalType
The base type of all Spark SQL data types.
- Since:
- 1.3.0
-
Constructor Summary
-
Method Summary
Modifier and TypeMethodDescriptionString representation for the type saved in external catalogs.abstract int
The default size of a value of this data type, used internally for size estimation.static boolean
equalsIgnoreCaseAndNullability
(DataType from, DataType to) Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.static boolean
equalsIgnoreNullability
(DataType left, DataType right) Compares two types, ignoring nullability of ArrayType, MapType, StructType.static boolean
equalsStructurally
(DataType from, DataType to, boolean ignoreNullability) Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.static boolean
equalsStructurallyByName
(DataType from, DataType to, scala.Function2<String, String, Object> resolver) Returns true if the two data types have the same field names in order recursively.static DataType
static DataType
json()
The compact JSON representation of this data type.static DataType
parseTypeWithFallback
(String schema, scala.Function1<String, DataType> parser, scala.Function1<String, DataType> fallbackParser) Parses data type from a string with schema.The pretty (i.e. indented) JSON representation of this data type.Readable string representation for the type.sql()
typeName()
Name of the type used in JSON serialization.
-
Constructor Details
-
DataType
public DataType()
-
-
Method Details
-
fromDDL
-
parseTypeWithFallback
public static DataType parseTypeWithFallback(String schema, scala.Function1<String, DataType> parser, scala.Function1<String, DataType> fallbackParser) Parses data type from a string with schema. It callsparser
forschema
. If it fails, callsfallbackParser
. If the fallback function fails too, combines error message fromparser
andfallbackParser
.- Parameters:
schema
- The schema string to parse byparser
orfallbackParser
.parser
- The function that should be invoke firstly.fallbackParser
- The function that is called whenparser
fails.- Returns:
- The data type parsed from the
schema
schema.
-
fromJson
-
equalsStructurally
Returns true if the two data types share the same "shape", i.e. the types are the same, but the field names don't need to be the same.- Parameters:
ignoreNullability
- whether to ignore nullability when comparing the typesfrom
- (undocumented)to
- (undocumented)- Returns:
- (undocumented)
-
equalsStructurallyByName
public static boolean equalsStructurallyByName(DataType from, DataType to, scala.Function2<String, String, Object> resolver) Returns true if the two data types have the same field names in order recursively.- Parameters:
from
- (undocumented)to
- (undocumented)resolver
- (undocumented)- Returns:
- (undocumented)
-
equalsIgnoreNullability
Compares two types, ignoring nullability of ArrayType, MapType, StructType.- Parameters:
left
- (undocumented)right
- (undocumented)- Returns:
- (undocumented)
-
equalsIgnoreCaseAndNullability
Compares two types, ignoring nullability of ArrayType, MapType, StructType, and ignoring case sensitivity of field names in StructType.- Parameters:
from
- (undocumented)to
- (undocumented)- Returns:
- (undocumented)
-
defaultSize
public abstract int defaultSize()The default size of a value of this data type, used internally for size estimation.- Returns:
- (undocumented)
-
typeName
Name of the type used in JSON serialization. -
json
The compact JSON representation of this data type. -
prettyJson
The pretty (i.e. indented) JSON representation of this data type. -
simpleString
Readable string representation for the type. -
catalogString
String representation for the type saved in external catalogs. -
sql
-