Spark SQL array functions

January 5, 2022

Spark SQL has a bunch of built-in functions, and many of them are geared towards arrays. For example, filter which filters an array using a predicate, and transform which maps an array using a function. Both the predicate and the function here use lambda syntax.

> SELECT filter(array(1, 2, 3), x -> x % 2 == 1);
 [1,3]
> SELECT filter(array(0, 2, 3), (x, i) -> x > i);
 [2,3]
> SELECT filter(array(0, null, 2, 3, null), x -> x IS NOT NULL);
 [0,2,3]

> SELECT transform(array(1, 2, 3), x -> x + 1);
 [2,3,4]
> SELECT transform(array(1, 2, 3), (x, i) -> x + i);
 [1,3,5]