In this post I am going to describe with example code as to how we can add a new column to an existing DataFrame using withColumn() function of DataFrame. There are 2 scenarios: The content of the new column is derived from the values of the existing column The new… Read more »
Jack and Daniel are friends. They want to encrypt their conversation so that they can save themselves from interception by a detective agency. So they invent a new cipher. Every message is encoded to its binary representation B of length N. Then it is written down K times, shifted by… Read more »
You will be given a list of 32 bits unsigned integers. You are required to output the list of the unsigned integers you get by flipping bits in its binary representation (i.e. unset bits must be set, and set bits must be unset). Take 1 for example, as unsigned 32-bits… Read more »
Earlier I was using Hive 0.13 and the query which was running fine in this version of Hive started misbehaving once I upgraded to Hive 1.2.My query was something looked something like below
|
1 2 3 |
SELECT count(*) as rows,userId FROM USERTABLE GROUP BY userId; |
Below is the snippet of the error.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
FailedPredicateException(identifier,{useSQL11ReservedKeywordsForIdentifier()}?) at org.apache.hadoop.hive.ql.parse.HiveParser_IdentifiersParser.identifier(HiveParser_IdentifiersParser.java:10937) at org.apache.hadoop.hive.ql.parse.HiveParser.identifier(HiveParser.java:47741) at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectItem(HiveParser_SelectClauseParser.java:2954) at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectList(HiveParser_SelectClauseParser.java:1414) at org.apache.hadoop.hive.ql.parse.HiveParser_SelectClauseParser.selectClause(HiveParser_SelectClauseParser.java:1141) at org.apache.hadoop.hive.ql.parse.HiveParser.selectClause(HiveParser.java:47833) at org.apache.hadoop.hive.ql.parse.HiveParser.selectStatement(HiveParser.java:42337) at org.apache.hadoop.hive.ql.parse.HiveParser.regularBody(HiveParser.java:42077) at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpressionBody(HiveParser.java:41260) at org.apache.hadoop.hive.ql.parse.HiveParser.queryStatementExpression(HiveParser.java:41130) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1622) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1122) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:202) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:531) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:326) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1273) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1327) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1210) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1200) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:242) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:194) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:409) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:340) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:442) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:488) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:843) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:795) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:706) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:223) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line 3:32 Failed to recognize predicate 'rows'. Failed rule: 'rows' in selection target |
Upon investigating I realized that “rows” is a… Read more »
The city of Gridland is represented as an matrix where the rows are numbered from to and the columns are numbered from to . Gridland has a network of train tracks that always run in straight horizontal lines along a row. In other words, the start and end points of… Read more »
A DataFrame is a collection of data, organized into named columns.DataFrames are similar to tables in a traditional database DataFrame can be constructed from sources such as Hive tables, Structured Data files, external databases, or existing RDDs. Under the hood, a DataFrame contains an RDD composed of Row objects with… Read more »
If we have a text file which contains numbers along with other data and we have to find the sum of those numbers , then AWK would come in handy. Ex:file.txt
|
1 2 3 4 |
10,Ajit 20,Sumit 22,Sam 35,john |
If the file contains above data , we could use AWK to find the sum of the number… Read more »
CombineByKey() is very similar to combiner of the mapreduce framework. In the MR framework, the combiner function is called in the map phase to do a local reduction and this value is then eventually sent over to reducer, this results in large savings if network bandwith. In SPARK, groupByKey() doesnt… Read more »