A Smooth Sea Never Made a Skillful Sailor

So I decided to start using more Python and less shell for operations and I realized the code I write on my dev machine, be it in my Arch Linux VM or on my Mac, will determine the final output of the script. Meaning, the same script I write on one platform may not work on the other. This is very frustrating because the code I write on my Mac expects v. 2.7.8 of Python and Arch may as well have a different version and there are major changes between minor versions of Python. Going with the same analogy, same script will not work with my Red Hat 6 machine because that has v. 2.6 of Python, really frustrating. I then decided to only write Python3 code, to my dismay, I have to jump through hoops to install Python3 on my Red Hat boxes. There is a software collection repository but I will have to maintain my own mirror for that, I really want to avoid that. So my questions to all is, how do you handle my situation? Thanks

DISCLAIMER: I do not offer any warranty for the below provided code, run at your own risk!!! UPDATE: Turns out I jumped the gun on the whole python script inside Oozie, it is absolutely possible, it's just in six hours of trial and error, I haven't found a solution yet! Oozie is at it's best, drains my life's blood and sanity. Either way, the shell script will work and it's been running fine for me for the last week. The Python script works on it's own but in the confines of Oozie, it doesn't know where the executable is. If you can get subprocess.Popen to work, shoot me a comment, I will greatly appreciate it. I have a love-hate relationship with Oozie. Truthfully, it's more hate than love though. Consider scenario, you need to create time-stamped directories, there are no built-in expression language functions available to just do it out of the box. You're forced to come up with all kinds of hacks to get what you want out of Oozie. What I use...

New release of my favorite terminal emulator MobaXterm is out with much needed fixes for latest security vulnerabilities as well as some new improvements like Cygwin repository support. I litteraly just found out so don't expect an in-depth overview, just go get it. If you want to read the release notes and download the latest version, go here .

I was just browsing through my past blog posts and noticed that all most recent blog entries were missing images. Did Google have an "oops" moment? I will try to locate the images and repost but most are unfortunately lost. Lesson learned: not to use Blogger as my documentation repository... Bummer!

This post is all about gadget envy. I just picked up a pair of bluetooth headphones following a Lifehacker deals post a few days ago. Here's the link to the post. I am by no means an audiophile, however, I was against getting bluetooth headphones for a long while. Recently, it dawned on me that BT headset is the only way to go for me when for the nth time my headphones stopped working in one ear. This deal is great and you may still have time to pick these up. Originally they're $80 and after a $50 off coupon provided at the link, these headphones are a steal for $30. They use the old 2.1 A2DP protocol but so far I haven't had any glaring issues with performance. Off the bat, the fit is very comfortable, something I've had issues with before with over the ear headphones. These do not snug tightly and don't make me feel like my head is in a vice. They come with a nice hard case, included is a 3.5mm cable and use the headphones just like any other wired set or use BT...

I was trying to execute some run-of-the-mill Mapreduce against HBase on a development cluster running HDP 2.1 and discovered some gotchas going from HDP 2.0 to HDP 2.1. For the unaware, HDP 2.0 stack is Hadoop 2.2.0 and HBase 0.96.0 and HDP 2.1 is Hadoop 2.4.0 and HBase 0.98.0. The stack I used was 2.1.3.0 and then desperately upgraded to the brand-new 2.1.4.0 stack with YARN bug fixes. Spoiler, that did not solve the actual problem. On a side-note, there's a lot of "coincidental" advice out-there and I was not able to fix my issues on following this link http://www.srccodes.com/p/article/46/noclassdeffounderror-org-apache-hadoop-service-compositeservice-shell-exitcodeexception-classnotfoundexception. This article did however put me on the right path, I went to the nodemanager where the failing task attempt executed and looked up the "real" error. The problem was class not found for one library that was moved from hadoop-common Maven artifact to hadoop-yarn-comm...

I just had a breakthrough moment when I realized why this error shows up when you run an Oozie workflow. We use Ambari for cluster management and by default Ambari has core-site.xml configured with these properties: The issue lies in the oozie.groups property. You need to make sure a user executing a workflow, must belong to "users" Linux group on the namenode server. Failsafe is definitely to have an asterisk for either property as most people recommend but I think this is a more granular approach. This idea dawned on me when I saw in the Ambari admin tab, the following: This means exactly that, user executing the workflow needs to belong to the proxy group controlled by hadoop.proxyuser.oozie.groups property. 

If you're running Isilon NAS and use it's Map/Reduce functionality for your workloads, you're probably still using Hadoop 1.x. If you're thinking of moving to Hadoop 2 and you have a secondary standalone cluster running Hadoop 2.2+ and you want to move data back and forth using utilities like distcp, I have bad news for you. Isilon does not support Hadoop 2.2. Sometime in the third quarter, they will release OneFS compatible with Hadoop 2.3+. The problem is with the underlying protobuf version incompatibility. There are some work-arounds available like doing distcp with webhdfs going from Isilon to standalone cluster but it doesn't work the other way around, at least I couldn't get it to work. On top of that, I'd lose packets during distcp via webhdfs and jobs would fail due to mismatched checksums. Great, so what is the solution, well you can also distcp using hftp protocol, it's a client independent protocol specifically built for incompatible hdfs cli...

Everyone is talking about security nowadays when it comes to Hadoop and it's ecosystem. Judging by the last two major acquisitions from Hortonworks and Cloudera , the major players are not taking it lightly either. I've been weary of security implications of maintaining an insecure Hadoop as well. Most Hadoop books dedicate a chapter or two on Hadoop security and up until now there were no books dedicated solely to Hadoop security. Choices were slim.. Enter Securing Hadoop . This book is only 120 pages and I was able to read it cover to cover on my commute to work in about a week. I will not provide a chapter by chapter summary of what this book offers, the book has a one page description of each chapter which describes everything better than I ever could. What I will say in this review is what this book does best and what it can improve on in the next iteration. This book is a "good to have" but not a "must have", unfortunately. The book does a good job at...

The other day I was working on MapReduce code over HBase tables and I discovered something really cool. Usually I'd have to package all HBase, Zookeeper, etc libraries else I'd get a ClassNotFoundException. I found this tip in HBase: Definitive Guide book. Apparently, if you specify scope "provided" in your Maven pom.xml file, Maven will not package the jars but it will expect that the jars are available on the cluster's classpath. I will save you my poor interpretation of this feature and point you to the Maven documentation. The feature is called Dependency Scope . This is how I define my dependencies now: So just to give you an idea, my jar size before adding this tag was 44Mb and after, it was 11Kb. Definitely saves time on transmitting the jars back and forth. Granted, this may not be a new tip to most people, I actually have seen this feature used when I was playing with Apache Storm, specifically the  storm-starter project but it never occurred to...

I've been looking for a comprehensive book on Apache Hadoop 2 and Yarn architecture, there are a few MEAPs available. This book in particular was finally released a few months back with all complete chapters. As all Hortonworks documentation, this book is well written and very easy to read. The choice to choose this book over others was simple. On top of that, it's written by the Hadoop committers so it's basically from the "horse's mouth". The current edition of the book has 12 chapters with additional material. The first chapter goes into history of how Hadoop came about and challenges the team at Yahoo had faced early in Hadoop history. This chapter opened up my eyes on how grandiose the project architecture was in the past and what it's become. It is very easy to take things for granted and this chapter does a great job explaining the choices the team had made. Chapter 2 gives a quick intro on how to deploy a single node cluster and start playing with ...

I found this recent paper mentioned in Storm mailing lists on yet another performance comparison of Streams and Storm. Giving credit to IBM, this time around it seems paper is written by developers and not sales people. Here's the direct link to the pdf. Most of the paper was based on v. 0.8 of Storm but at the end of the paper comparison with Storm 0.9.0.1 was also referenced. Comparison was done using Storm 0.8 with ZeroMQ and Storm 0.9 with Netty for transfer protocol. It is an interesting read for a change. I am also surprised to see Apache Avro used for serialization. I will not cloud your judgement by stating my opinion but I remain skeptical of these papers. I urge Storm community to offer its findings from own comparison. One thing I'd like to state is that again IBM claims it is much faster to implement a use-case using Streams over Storm and from my own experience, I was able to install Storm 0.8, configure my IDE to develop and test topologies, implement my use case...

IBM has a white paper from 2013 comparing Streams with Storm. As expected, the paper is full of marketing mambo jumbo applicable for suits. I usually try to avoid such information but I couldn't resist. I have to give IBM credit for even having such a paper, to acknowledge Storm as a market leader, even if the motive is somewhat shady. The paper is a bit outdated, stating Storm is GPL licensed software and has no market leading companies behind it. If you haven't heard, Hortonworks has picked up Storm and has some committers dedicated to it. It's also part of Hortonworks Data Platform stack as of v. 2.1. In addition to that, Storm is now a top level Apache project and no longer GPL. 0MQ messaging is now a second class citizen in favor of Netty, a fully native Java stack. The argument of the paper is that Storm lacks enterprise support. Hortonworks will gladly provide it. Either way, this paper is kind of expected from a large vendor like IBM. I'm in no role suitable to ...

I just found out about a RHEL initiative to offer "software collections" which to me seems similar to Ubuntu's private repositories. From the announcement "Software Collections allow you to build and install applications without being constrained by the default versions that are installed on your system. Say you want to deploy an application that requires MariaDB 5.5 and PHP 5.4 or Python 3.3 on a system running CentOS 6.5. You don’t want to have to deploy a new OS. You don’t want to compile your own, and you may have other applications that depend on the system versions of those components." This certainly offers a very flexible option to install software on your RHEL servers. There is of course EPEL but this comes with the official RHEL support. This solves a number of my problems. Security audits always flag httpd packages on my servers as insecure just by looking at the major versions. Yes, there are ways to limit flags in our audit tools but it's not...

From the site: "MobaXterm is an enhanced terminal for Windows with an X11 server, a tabbed SSH client and several other network tools for remote computing (VNC, RDP, telnet, rlogin). MobaXterm brings all the essential Unix commands to Windows desktop, in a single portable exe file which works out of the box". In the past few years of working with Open Source and Hadoop, I've come to rely on terminal emulators to interact with my servers. You have many options and I've used Cygwin and Console2 before but ever since I've come across MobaXterm, I haven't looked back. MobaXterm is a very stable terminal emulator with SSH. It includes a bunch of additional features like multiexec, which duplicates your terminal activity in additional tabs, so in other words if you'd like to execute the same command on multiple servers, SSH to all the servers and start typing out your commands, it will just work. It supports extensions, Git, SVN and a bunch more are available, ...

We're always looking at improving our realtime processing and over the past six months or so I've been looking at Apache Storm and IBM Streams. There are a few others like LinkedIn's Samza and Spark Streaming but I haven't had a chance to look into them. I will leave my findings for another post but for now, I have great news for people getting started with Streams. IBM began open-sourcing their Streams toolkits. Here's the announcement and links for those interested. The other most awesome news as far as I'm concerned is that IBM Streams is now able to run within a YARN application. I mean, with Apache Slider project announcement recently this would've been a natural thing but I'm glad IBM released it even before any work on Slider was done and Streams is compatible with Hadoop 2.2. This is definitely a step in the right direction if Streams wants a piece of Storm's marketshare. I am genuinely excited about the news. On a side note, I'm going...

If you can read this and it successfully redirects to http://blog.ervits.com then I've ported my blogger blog to my own domain correctly :). Currently it says I need to wait 24hrs to correctly redirect, so fingers crossed. In the last few years I've moved on from relational world and have embrased NoSQL. For the past two years I've been working with Hadoop, HBase and other tools in the ecosystem. I am going to share my discoveries along the way so stay tuned.. Thanks UPDATE: fixed redirect for the blog