Author Archives: baahu

SPARK :Add a new column to a DataFrame using UDF and withColumn()

In this post I am going to describe with example code as to how we can add a new column to an existing DataFrame using withColumn() function of DataFrame. There are 2 scenarios: The content of the new column is derived from the values of the existing column The new… Read more »

Hive Error:Failed to recognize predicate Failed rule: ‘identifier’ column specification

Earlier I was using Hive 0.13 and the query which was running fine in this version of Hive started misbehaving once I upgraded to Hive 1.2.My query was something looked something like below SELECT count(*) as rows,userId FROM USERTABLE GROUP BY userId; Below is the snippet of the error. FailedPredicateException(identifier,{useSQL11ReservedKeywordsForIdentifier()}?)… Read more »