Identify failing machines and missing clients for Oozie workflows

If you ever come across an issue where your Oozie workflow works on most nodes but fails on others, a neat trick to identify a missing client would be to put the following in your shell action script.
echo "I AM: `whoami`"
echo "Running On: `hostname`"
echo "CWD: `pwd`"
echo "Can I see these clients? Hive = `which hive`, Sqoop = `which sqoop`"
echo "`ls -l /usr/hdp/current/sqoop-client/ | head -n 5`"
in my case, I was looking for what user was executing the shell action, in this case it was “yarn” user and it was running on a hostname identified by the hostname command. From then on, it was an easy fix to install the missing client on that host and workflow worked again. Feel free to add other commands to the script to fit your use case. I was testing whether sqoop and hive clients were installed, I also have a check whether /usr/hdp/current/sqoop-client directory exists.

I have an example shell action on my github.

Comments

Post a Comment

Popular posts from this blog

Vista Vulnerability Report mentions Ubuntu 6.06 LTS

Update: I posted this on MIXX since DIGG was taken care of by someone already, except that that user didn't really write a good preview for it so it got buried. This report is generating some buzz though. I emailed this report to Mark Shuttleworth of Ubuntu fame and little did I know, HE ANSWERED!!!! Whoever answered me on that side of the e-mail pipe liked the report and was wondering if it was official and whether he could quote from it. It definitely is official and I forwarded him the e-mail with this report, he said he will pass it down internally. That made my day, it's amazing that a man of his statute, that's if it was him, answers e-mails from regular people like me. That's what I call a real genuine pioneer! Thank you Mark and all of the Ubuntu Staff. Read
Read more

Running CockroachDB with Docker Compose and Minio, Part 2

This is my second post on creating a multi-service architecture with docker-compose. This is meant to be a learning exercise and typically docker-compose is used to set up a local development environment rather than a production-ready set up. I regularly, find myself building these environments to reproduce customer bugs. For a production-specific application, refer to your platform vendor documentation. At some later time, I will cover Kubernetes deployments that can be used as a stepping stone for a real-world application. Until then, let's focus on the task at hand. We're building a microservice architecture with CockroachDB writing changes in real-time to an S3 bucket in JSON format. S3 bucket is served by a service called Minio. It can act like an S3 appliance on premise or serve as a local gateway to your cloud storage. Let's dig in:
Read more

Doing "print screen" on a Mac is a pain in the ass

I am catching up with blogging today and I need to take a screenshot of my screen. I have a Macbook Pro with Ubuntu in a virtual machine. There is no printscreen button and a quick search returned "Apple + Shift + 3". Mac fanboys think that's easy and better than PC? So here's a problem. If you want to take a print screen in VM, that key combination doesn't work. What do I do besides opening the screenshot application and doing it half-assed way? I am sure you can edit key bindings but that should not be that way. If you know of any other ways, let me know.
Read more