running HCatalog commands within Pig scripts in Oozie


Apache Pig introduced a new option of executing HCatalog commands inside Pig scripts and Grunt shell. For details to get started with it, take a look at my previous post.
When you try to execute a pig script with HCatalog command in Oozie, you will actually get the same error, irrelevant whether you changed pig.properties file as described in the previous article. The error is below:
ERROR 2997: Encountered IOException. hcat.bin is not defined. Define it to be your hcat script (Usually $HCAT_HOME/bin/hcat

java.io.IOException: hcat.bin is not defined. Define it to be your hcat script (Usually $HCAT_HOME/bin/hcat
 at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1283)
 at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501)
 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
 at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
 at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
 at org.apache.pig.Main.run(Main.java:502)
 at org.apache.pig.PigRunner.run(PigRunner.java:49)
 at org.apache.oozie.action.hadoop.PigMain.runPigJob(PigMain.java:288)
 at org.apache.oozie.action.hadoop.PigMain.run(PigMain.java:231)
 at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:47)
 at org.apache.oozie.action.hadoop.PigMain.main(PigMain.java:76)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:606)
 at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:236)
 at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
 at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
 at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:415)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
================================================================================
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.PigMain], exit code [2]

 To make this work with Oozie, you need to do a couple of things:
make sure these properties exist in job.properties
oozie.use.system.libpath=true
hcatNode=thrift://sandbox.hortonworks.com:9083
db=default
table=sample08
Then in the beginning of your script, add the following:
set hcat.bin /usr/bin/hcat;
In your workflow.xml, specify the script and add file property pointing to lib/hive-site.xml



then create a directory called lib and place your hive-site.xml file in it.
your workflow directory tree should look similar to this, README is optional

I have a sample workflow on my github.

Comments

Post a Comment

Popular posts from this blog

Vista Vulnerability Report mentions Ubuntu 6.06 LTS

Update: I posted this on MIXX since DIGG was taken care of by someone already, except that that user didn't really write a good preview for it so it got buried. This report is generating some buzz though. I emailed this report to Mark Shuttleworth of Ubuntu fame and little did I know, HE ANSWERED!!!! Whoever answered me on that side of the e-mail pipe liked the report and was wondering if it was official and whether he could quote from it. It definitely is official and I forwarded him the e-mail with this report, he said he will pass it down internally. That made my day, it's amazing that a man of his statute, that's if it was him, answers e-mails from regular people like me. That's what I call a real genuine pioneer! Thank you Mark and all of the Ubuntu Staff. Read
Read more

Running CockroachDB with Docker Compose and Minio, Part 2

This is my second post on creating a multi-service architecture with docker-compose. This is meant to be a learning exercise and typically docker-compose is used to set up a local development environment rather than a production-ready set up. I regularly, find myself building these environments to reproduce customer bugs. For a production-specific application, refer to your platform vendor documentation. At some later time, I will cover Kubernetes deployments that can be used as a stepping stone for a real-world application. Until then, let's focus on the task at hand. We're building a microservice architecture with CockroachDB writing changes in real-time to an S3 bucket in JSON format. S3 bucket is served by a service called Minio. It can act like an S3 appliance on premise or serve as a local gateway to your cloud storage. Let's dig in:
Read more

Doing "print screen" on a Mac is a pain in the ass

I am catching up with blogging today and I need to take a screenshot of my screen. I have a Macbook Pro with Ubuntu in a virtual machine. There is no printscreen button and a quick search returned "Apple + Shift + 3". Mac fanboys think that's easy and better than PC? So here's a problem. If you want to take a print screen in VM, that key combination doesn't work. What do I do besides opening the screenshot application and doing it half-assed way? I am sure you can edit key bindings but that should not be that way. If you know of any other ways, let me know.
Read more