A Smooth Sea Never Made a Skillful Sailor

it is not completely obvious but you can certainly run Python scripts within Here’s a sample job.properties file, nothing special about it. nameNode=hdfs://sandbox .hortonworks .com : 8020 jobTracker=sandbox .hortonworks .com : 8050 queueName=defaultexamplesRoot=oozie oozie .wf .application .path =${nameNode}/user/${user .name }/${examplesRoot}/apps/python Here’s a sample workflow that will look for a script called script.py inside scripts folder < workflow-app xmlns = "uri:oozie:workflow:0.4" name = "python-wf" > < start to = "python-node" /> < action name = "python-node" > < shell xmlns = "uri:oozie:shell-action:0.2" > < job-tracker > ${jobTracker} </ job-tracker > < name-node > ${nameNode} </ name-node > < configuration > < property > < name > mapred.j...

O'Reilly recently released a free ebook called Hadoop with Python by the author of MapReduce Design Patterns, Donald Miner. Needless to say that caught my eye. The book is a short read, I was able to run through it within two lunch hours. It has five chapters tackling different angles of Hadoop. It is an easy read with an excellent overview of each product discussed. 1st chapter discusses HDFS and Spotify's library written in Python called Snakebite that allows for Python shops interact with HDFS in a native way. This is pretty use-case specific because I don't see a reason to use the library unless you're a Python-heavy shop. The other drawback is that it's not Python3 compliant. That may be an issue going forward. The cool think about Snakebite is that the library does not require loading any Java libraries and promises to be really fast to load. It leverages RPC to speak to Namenode and uses protobuf, so interaction is native. 2nd chapter is on writing MapR...

If you ever come across an issue where your Oozie workflow works on most nodes but fails on others, a neat trick to identify a missing client would be to put the following in your shell action script. echo "I AM: `whoami`" echo "Running On: `hostname`" echo "CWD: `pwd`" echo "Can I see these clients? Hive = `which hive`, Sqoop = `which sqoop`" echo "`ls -l /usr/hdp/current/sqoop-client/ | head -n 5`" in my case, I was looking for what user was executing the shell action, in this case it was “yarn” user and it was running on a hostname identified by the hostname command. From then on, it was an easy fix to install the missing client on that host and workflow worked again. Feel free to add other commands to the script to fit your use case. I was testing whether sqoop and hive clients were installed, I also have a check whether /usr/hdp/current/sqoop-client directory exists. I have an example shell action on my github .

This is yet another one of those episodes where it seems it should be an obvious thing to do in Oozie and it isn’t. The use case is to run an undefined number of sqoop import commands from previously unknown list of tables. The one advice you’d hear is to create a sqoop action for each table. This unfortunately doesn’t support the use case. Let’s assume you have no access to source database and tables. You need to sqoop every table dynamically from a database, list of tables changes from week to week. A static workflow will not work. The one solution that I can think of is to create a shell script, loop through each table and run sqoop against each table. Not so fast! The issue here is that the user executing a workflow of type shell action is “yarn” and sqoop command is executed by the original user executing the workflow. So let’s say user is “root”, it executes a workflow with one shell action that calls command “sqoop import --connect jdbc:postgresql://127.0.0.1/ambari --username ...

Apache Pig introduced a new option of executing HCatalog commands inside Pig scripts and Grunt shell. For details to get started with it, take a look at my previous post . When you try to execute a pig script with HCatalog command in Oozie, you will actually get the same error, irrelevant whether you changed pig.properties file as described in the previous article. The error is below: ERROR 2997: Encountered IOException. hcat.bin is not defined. Define it to be your hcat script (Usually $HCAT_HOME/bin/hcat java.io.IOException: hcat.bin is not defined. Define it to be your hcat script (Usually $HCAT_HOME/bin/hcat at org.apache.pig.tools.grunt.GruntParser.processSQLCommand(GruntParser.java:1283) at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:501) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230) at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205) at org.apache.pig.to...

If you're running HDP 2.3.0 or 2.3.2 and you're eager to try calling HCatalog commands in your Pig scripts there is a gotcha that you need to be aware of. Apache Pig recently introduced an option of calling HCatalog and Hive commands within Pig. For example, assume we have a file called file.pig. Where file.pig is a regular pig script but it contains the following statement: sql show tables; This will actually work in Sandbox and display the existing tables. You can follow that with your typical Pig commands. However, If you your vanilla cluster or Sandbox is not modified with changes below, you will get the following error: Pig Stack Trace --------------- ERROR 2997: Encountered IOException. /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. java.io.IOException: /usr/local/hcat/bin/hcat does not exist. Please check your 'hcat.bin' setting in pig.properties. at org.apache.pig.tools.grunt.GruntParser....