Friday, October 23, 2015

fix for error "hcat.bin is not defined. Define it to be your hcat script" on HDP 2.3.0 and 2.3.2

If you're running HDP 2.3.0 or 2.3.2 and you're eager to try calling HCatalog commands in your Pig scripts there is a gotcha that you need to be aware of.
Apache Pig recently introduced an option of calling HCatalog and Hive commands within Pig. For example, assume we have a file called file.pig.
Where file.pig is a regular pig script but it contains the following statement:

sql show tables;

This will actually work in Sandbox and display the existing tables. You can follow that with your typical Pig commands. However, If you your vanilla cluster or Sandbox is not modified with changes below, you will get the following error:

Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1286) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81) at org.apache.pig.Main.run(Main.java:631) at org.apache.pig.Main.main(Main.java:177) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 

The problem is that pig.properties in /etc/pig/conf/ is set to /usr/local/hcat/bin/hcat by default. To fix the problem, you have at least four options that I can think of. 

1. Go to Ambari > Configs > Advanced pig.properties and change hcat.bin to the following:

hcat.bin=/usr/bin/hcat 

Then restart pig clients 

2. Copy the pig.properties file to your home directory and change the hcat.bin to the same as in 1. Execute your script like so: 

pig -P pig.properties file.pig

3. Override the property on the fly

pig -Dhcat.bin=/usr/bin/hcat file.pig

4. Put the following in your pig script

set hcat.bin /usr/bin/hcat;


Post a Comment