Hi,
I'm trying to understand why comparing an int with a long in a bincond fails in some cases but does not in others.
Here's the sample.csv dataset:
a,5
a,7
a,6
b,3
b,9
Case 1:
A = load 'sample.txt' using PigStorage(',') as (id:chararray, val:int);
B = foreach A generate id, (case when val < 2 THEN 'less than 2'
ELSE 'More than or equal to 2'
END);
dump B;
This succeeds.
Case 2:
A = load 'sample.txt' using PigStorage(',') as (id:chararray, val:int);
C = group A by custid;
D = foreach C generate COUNT(A) as cnt:int, flatten(A);
E = foreach D generate id, (case when cnt < 2 THEN 'less than 2'
ELSE 'More than or equal to 2'
END);
dump E;
This fails with the following exceptions:
(I know this can be fixed by assigning "cnt"'s schema as "long" but trying to understand why case 2 failed when case 1 didn't.
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias E. Backend error : Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias E. Backend error : Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at org.apache.pig.PigServer.openIterator(PigServer.java:872)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:382)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:302)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:464)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:412)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at java.lang.Long.compareTo(Long.java:50)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LessThanExpr.doComparison(LessThanExpr.java:109)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LessThanExpr.getNextBoolean(LessThanExpr.java:79)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.genericGetNext(POBinCond.java:72)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNextString(POBinCond.java:138)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:315)
Gufran Pathan| +91 7760913355| www.mu-sigma.com<http://www.mu-sigma.com/> |
Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing "look over there." -Randall Munroe
Disclaimer: http://www.mu-sigma.com/disclaimer.html
I'm trying to understand why comparing an int with a long in a bincond fails in some cases but does not in others.
Here's the sample.csv dataset:
a,5
a,7
a,6
b,3
b,9
Case 1:
A = load 'sample.txt' using PigStorage(',') as (id:chararray, val:int);
B = foreach A generate id, (case when val < 2 THEN 'less than 2'
ELSE 'More than or equal to 2'
END);
dump B;
This succeeds.
Case 2:
A = load 'sample.txt' using PigStorage(',') as (id:chararray, val:int);
C = group A by custid;
D = foreach C generate COUNT(A) as cnt:int, flatten(A);
E = foreach D generate id, (case when cnt < 2 THEN 'less than 2'
ELSE 'More than or equal to 2'
END);
dump E;
This fails with the following exceptions:
(I know this can be fixed by assigning "cnt"'s schema as "long" but trying to understand why case 2 failed when case 1 didn't.
Pig Stack Trace
ERROR 1066: Unable to open iterator for alias E. Backend error : Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias E. Backend error : Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at org.apache.pig.PigServer.openIterator(PigServer.java:872)
at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:774)
at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:69)
at org.apache.pig.Main.run(Main.java:541)
at org.apache.pig.Main.main(Main.java:156)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: org.apache.pig.backend.executionengine.ExecException: ERROR 0: Exception while executing [POBinCond (Name: POBinCond[chararray] - scope-298 Operator Key: scope-298) children: [[LessThanExpr (Name: Less Than[boolean] - scope-295 Operator Key: scope-295) children: [[POProject (Name: Project[int][0] - scope-293 Operator Key: scope-293) children: null at []], [ConstantExpression (Name: Constant(10) - scope-294 Operator Key: scope-294) children: null at []]] at []], [ConstantExpression (Name: Constant(less than 20) - scope-296 Operator Key: scope-296) children: null at []], [ConstantExpression (Name: Constant(More than or equal to 20) - scope-297 Operator Key: scope-297) children: null at []]] at []]: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:338)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:382)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNextTuple(POForEach.java:302)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.runPipeline(PigGenericMapReduce.java:464)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.processOnePackageOutput(PigGenericMapReduce.java:432)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:412)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Reduce.reduce(PigGenericMapReduce.java:256)
at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1594)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
Caused by: java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
at java.lang.Long.compareTo(Long.java:50)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LessThanExpr.doComparison(LessThanExpr.java:109)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.LessThanExpr.getNextBoolean(LessThanExpr.java:79)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.genericGetNext(POBinCond.java:72)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POBinCond.getNextString(POBinCond.java:138)
at org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:315)
Gufran Pathan| +91 7760913355| www.mu-sigma.com<http://www.mu-sigma.com/> |
Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing "look over there." -Randall Munroe
Disclaimer: http://www.mu-sigma.com/disclaimer.html