Friday, April 17, 2015

Hadoop 2.6 move to another queue feature

this is a quick post about Hadoop 2.6 that I just discovered. I'm finishing up with HDP 2.2 upgrade and I just found out that I can switch running applications between YARN queues. This is an incredibly useful feature. We sometimes issues jobs w/out specifying the designated queues, pushing the job to the right queue is immensely useful.
yarn application -movetoqueue applicationID -queue queuename

Saturday, March 28, 2015

Sort a set of strings by each string's last character, solution 2

Here's another solution to the same problem, this time I am not using a comparator. Instead, I reverse each element, save it in a new TreeSet which keeps elements in sorted order, then instantiate a new LinkedHashSet which maintains an insertion order, reverse the strings again to the way there were. What I noticed doing this problem is that this solution is faster with larger sample size than solution with comparator. That's a surprising revelation. Also, using StringBuilder to reverse a string is a lot faster than reversing character by character.
 package com.ervits;  
 import java.util.Arrays;  
 import java.util.HashSet;  
 import java.util.LinkedHashSet;  
 import java.util.Objects;  
 import java.util.Set;  
 import java.util.TreeSet;  
 /**  
  *  
  * @author Artem  
  * @created Mar 27, 2015 2:15:49 PM  
  *  
  * Project SetSorter  
  *  
  */  
 public class SetSorterWithReverseString {  
   public static void main(String[] args) {  
     long start = System.currentTimeMillis();  
     Set<String> unsorted = new HashSet<>();  
     unsorted.addAll(Arrays.asList("Abstract", "class", "is", "a", "special",   
         "class", "in", "Java", "it", "can", "not", "be", "instantiated",   
         "and", "that's", "why", "can", "not", "be", "used", "directly",  
         "At", "first", "concept", "of", "abstraction", "abstract",   
         "class", "and", "interface", "all", "look", "useless", "to",   
         "many", "developers", "because", "you", "can", "not", "implement",   
         "any", "method", "in", "interface", "you", "can", "not", "create", "object",  
         "To", "create", "these", "etexts", "the", "Project", "expends", "considerable",  
         "efforts", "to", "identify", "transcribe", "and", "proofread", "public", "domain",  
         "Take", "the", "hue", "of", "the", "company", "you", "are", "with", "Take",   
         "characters", "as", "they", "do", "most", "things", "upon", "trust",   
         "Take", "rather", "than", "give", "the", "tone", "of", "the", "company",   
         "you", "are", "in", "Take", "nothing", "for", "granted", "upon", "the",   
         "bare", "authority", "of", "the", "author", "Taking", "up", "adventitious",   
         "proves", "their", "want", "of", "intrinsic", "merit", "Talent", "of", "hating",   
         "with", "good_breeding", "and", "loving", "with", "prudence", "Talk", "often",   
         "but", "never", "long", "Talk", "sillily", "upon", "a", "subject", "of", "other",   
         "people's", "Talk", "of", "natural", "affection", "is", "talking", "nonsense",   
         "Talking", "of", "either", "your", "own", "or", "other", "people's", "domestic",   
         "affairs", "Tell", "me", "whom", "you", "live", "with", "and", "I", "will",   
         "tell", "you", "who", "you", "are", "Tell", "stories", "very", "seldom", "The",   
         "longest", "life", "is", "too", "short", "for", "knowledge", "The", "present",   
         "moments", "are", "the", "only", "ones", "we", "are", "sure", "of", "The", "best",   
         "have", "something", "bad", "and", "something", "little", "The", "worst", "have",   
         "something", "good", "and", "sometimes", "something", "great", "There", "are",   
         "many", "avenues", "to", "every", "man", "They", "thought", "I", "informed",   
         "because", "I", "pleased", "them", "Thin", "veil", "of", "Modesty", "drawn",   
         "before", "Vanity", "Think", "to", "atone", "by", "zeal", "for", "their",   
         "want", "of", "merit", "and", "importance", "Think", "yourself", "less",   
         "well", "than", "you", "are", "in", "order", "to", "be", "quite", "so",   
         "Thinks", "himself", "much", "worse", "than", "he", "is", "Thoroughly",   
         "not", "superficially", "Those", "who", "remarkably", "affect", "any",   
         "one", "virtue", "Those", "whom", "you", "can", "make", "like", "themselves",   
         "better", "Three", "passions", "that", "often", "put", "honesty", "to",   
         "most", "severe", "trials", "Timidity", "and", "diffidence", "To", "be",   
         "heard", "with", "success", "you", "must", "be", "heard", "with", "pleasure",   
         "To", "be", "pleased", "one", "must", "please", "To", "govern", "mankind",   
         "one", "must", "not", "overrate", "them", "To", "seem", "to", "have", "forgotten",   
         "what", "one", "remembers", "To", "know", "people's", "real", "sentiments",   
         "I", "trust", "much", "more", "to", "my", "eyes", "To", "great", "caution",   
         "you", "can", "join", "seeming", "frankness", "and", "openness", "Too", "like",   
         "and", "too", "exact", "a", "picture", "of", "human", "nature", "Trifle", "only",   
         "with", "triflers", "and", "be", "serious", "only", "with", "the", "serious",   
         "Trifles", "that", "concern", "you", "are", "not", "trifles", "to", "me",   
         "Trifling", "parts", "with", "their", "little", "jargon", "Trite", "jokes",   
         "and", "loud", "laughter", "reduce", "him", "to", "a", "buffoon", "Truth",   
         "but", "not", "the", "whole", "truth", "must", "be", "the", "invariable",   
         "principle", "Truth", "leaves", "no", "room", "for", "compliments", "Unaffected",   
         "silence", "upon", "that", "subject", "is", "the", "only", "true", "medium",   
         "Unguarded", "frankness", "Unintelligible", "to", "his", "readers", "and",   
         "sometimes", "to", "himself", "Unopened", "because", "one", "title", "in",   
         "twenty", "has", "been", "omitted", "Unwilling", "and", "forced", "it",   
         "will", "never", "please", "Use", "palliatives", "when", "you", "contradict",   
         "Useful", "sometimes", "to", "see", "the", "things", "which", "one", "ought",   
         "to", "avoid", "Value", "of", "moments", "when", "cast", "up", "is", "immense",   
         "Vanity", "interest", "and", "absurdity", "always", "display", "Vanity", "that",   
         "source", "of", "many", "of", "our", "follies", "Warm", "and", "young", "thanks",   
         "not", "old", "and", "cold", "ones", "Water_drinkers", "can", "write", "nothing",   
         "good", "We", "love", "to", "be", "pleased", "better", "than", "to", "be",   
         "informed", "We", "have", "many", "of", "those", "useful", "prejudices", "in",   
         "this", "country", "We", "shall", "be", "feared", "if", "we", "do", "not",   
         "show", "that", "we", "fear", "Well", "dressed", "not", "finely", "dressed",   
         "What", "pleases", "you", "in", "others", "will", "in", "general", "please",   
         "them", "in", "you", "What", "displeases", "or", "pleases", "you", "in",   
         "others", "What", "you", "feel", "pleases", "you", "in", "them", "What",   
         "have", "I", "done", "to_day", "What", "is", "impossible", "and", "what",   
         "is", "only", "difficult", "Whatever", "pleases", "you", "most", "in",   
         "others", "Whatever", "is", "worth", "doing", "at", "all", "is", "worth",   
         "doing", "well", "Whatever", "one", "must", "do", "one", "should", "do",   
         "de", "bonne", "grace", "Whatever", "real", "merit", "you", "have",   
         "other", "people", "will", "discover", "When", "well", "dressed",   
         "for", "the", "day", "think", "no", "more", "of", "it", "afterward",   
         "Where", "one", "would", "gain", "people", "remember", "that", "nothing",   
         "is", "little", "Who", "takes", "warning", "by", "the", "fate", "of",   
         "others", "Wife", "very", "often", "heard", "indeed", "but", "seldom",   
         "minded", "Will", "not", "so", "much", "as", "hint", "at", "our", "follies",   
         "Will", "pay", "very", "dear", "for", "the", "quarrels", "and", "ambition",   
         "of", "a", "few", "Wish", "you", "my", "dear", "friend", "as", "many",   
         "happy", "new", "years", "as", "you", "deserve", "Wit", "may", "created",   
         "any", "admirers", "but", "makes", "few", "friends", "Witty", "without",   
         "satire", "or", "commonplace", "Woman", "like", "her", "who", "has",   
         "always", "pleased", "and", "often", "been", "pleased", "Women", "are",   
         "the", "only", "refiners", "of", "the", "merit", "of", "men", "Women",   
         "choose", "their", "favorites", "more", "by", "the", "ear", "Women",   
         "are", "all", "so", "far", "Machiavelians", "Words", "are", "the",   
         "dress", "of", "thoughts", "World", "is", "taken", "by", "the",   
         "outside", "of", "things", "Would", "not", "tell", "what", "she",   
         "did", "not", "know", "Wrapped", "up", "and", "absorbed", "in",   
         "their", "abstruse", "speculations", "Writing", "anything", "that",   
         "may", "deserve", "to", "be", "read", "Writing", "what", "may",   
         "deserve", "to", "be", "read", "Wrongs", "are", "often",   
         "forgiven", "but", "contempt", "never", "is", "Yielded", "commonly",   
         "without", "conviction", "You", "must", "be", "respectable", "if",   
         "you", "will", "be", "respected", "You", "had", "much", "better",   
         "hold", "your", "tongue", "than", "them", "Young", "people", "are",   
         "very", "apt", "to", "overrate", "both", "men", "and", "things",   
         "Young", "fellow", "ought", "to", "be", "wiser", "than", "he",   
         "should", "seem", "to", "be", "Young", "men", "are", "as", "apt",   
         "to", "think", "themselves", "wise", "enough", "Your", "merit",   
         "and", "your", "manners", "can", "alone", "raise", "you", "Your",   
         "character", "there", "whatever", "it", "is", "will", "get",   
         "before", "you", "here"));  
     Set<String> sorted = reverseAndSort(unsorted);  
     System.out.println("***** UNSORTED *****");  
     System.out.println(unsorted.toString());  
     System.out.println("***** SORTED *****");  
     System.out.println(sorted.toString());  
     long end = System.currentTimeMillis();  
     System.out.printf("Time taken: %d\n", (end - start));  
     System.out.printf("Unsorted size: %d\n", unsorted.size());  
     System.out.printf("Sorted size: %d\n", sorted.size());  
   }  
   public static Set<String> reverseAndSort(Set<String> unsorted) {  
     Set<String> sorted = new TreeSet<>();  
     for (String aStr : unsorted) {  
       if(aStr != null) {  
         sorted.add(reverseWithStringBuilder(aStr));  
       }    
     }  
     return normalizeSet(sorted);  
   }  
   static String reverseString(String aStr) {  
     String reversed = "";  
     for (int i = aStr.length() - 1; i >= 0; i--) {  
       reversed += aStr.charAt(i);  
     }  
     return reversed;  
   }  
   static Set<String> normalizeSet(Set<String> set) {  
     Set<String> finalSet = new LinkedHashSet<>();  
     Objects.requireNonNull(set, "passed set is null");  
     for (String aStr : set) {  
       if(aStr != null) {  
         finalSet.add(reverseWithStringBuilder(aStr));  
       }  
     }  
     return finalSet;  
   }  
   // Faster than reverseString()  
   static String reverseWithStringBuilder(String aStr) {  
     Objects.requireNonNull(aStr, "passed String cannot be null");  
     return new StringBuilder(aStr).reverse().toString();  
   }  
 }  

Wednesday, March 25, 2015

Sort a set of strings by each string's last character, solution using comparator (Update)

I have the following problem. Sort a set of strings by each string's last character. This is my implementation, please suggest other solutions.

UPDATE
turns out I had a bug in my previous version, I was only comparing the last character to last character and on the return from the method, it would ignore every other word with same last character. It was only noticeable after a much larger sample set.



 package com.ervits;  
 import java.util.Comparator;  
 import java.util.Objects;  
 /**  
  *  
  * @author Artem  
  * @created Mar 27, 2015 2:07:21 PM  
  *  
  * Project SetSorter  
  *  
  */  
 class LastCharComparator implements Comparator {  
   public LastCharComparator() {  
   }  
   @Override  
   public int compare(String t1, String t2) {  
     Objects.requireNonNull(t1, "passed String1 cannot be null");  
     Objects.requireNonNull(t2, "passed String2 cannot be null");  
     final int result;  
     if(reverseWithStringBuilder(t1)  
         .compareTo(reverseWithStringBuilder(t2)) &gt; 0) {  
       result = 1;  
     } else if (reverseWithStringBuilder(t1)  
         .compareTo(reverseWithStringBuilder(t2)) &lt; 0) {  
       result = -1;  
     } else {  
       result = 0;  
     }  
     return result;  
   }  
   static String reverseWithStringBuilder(String aStr) {  
     return new StringBuilder(aStr).reverse().toString();  
   }  
 }  
 package com.ervits;  
 import java.util.Arrays;  
 import java.util.Comparator;  
 import java.util.LinkedHashSet;  
 import java.util.Set;  
 import java.util.SortedSet;  
 import java.util.TreeSet;  
 /**  
  *  
  * @author artem  
  */  
 public class SetSorter {  
   public static void main(String[] args) {  
     Set unsorted = new LinkedHashSet&lt;&gt;();  
     unsorted.addAll(Arrays.asList("xxxxxX", "xxxxxY", "hell7oD", "hel1loB", "helloC", "helloA", "helloE", "helloA", "helloZ"));  
     Comparator setComparator = new LastCharComparator();  
     SortedSet sorted = new TreeSet&lt;&gt;(setComparator);  
     sorted.addAll(unsorted);  
     System.out.println("***** UNSORTED *****");  
     System.out.println(unsorted.toString());  
     System.out.println("***** SORTED *****");  
     System.out.println(sorted.toString());  
     System.out.println("***** COMPARATOR *****");  
     System.out.println(sorted.comparator().getClass().getCanonicalName());  
   }  
 }  

Thursday, March 5, 2015

Running saved sqoop 1.4.4 jobs with password file

This has been bugging me for a long time. I've been trying to pass password to sqoop job securely for the longest time with no avail. Apparently, my approach was flawed by two things. First of all, the sqoop user guide 1.4.4 has incorrect spelling for password file option in the example. The documentation has the correct option but if you're like me and follow the example you will get it wrong.

wrong way in Sqoop 1.4.4 user guide:
sqoop import --connect jdbc:mysql://database.example.com/employees \
    --username venkatesh --passwordFile ${user.home}/.password

right way in Sqoop 1.4.5 user guide:
sqoop import --connect jdbc:mysql://database.example.com/employees \
    --username venkatesh --password-file ${user.home}/.password

the other flaw is that when you create the password file with "echo" command, make sure you don't append a new line character to the password. Here's a handy script to create a password:

echo -n "password" > .password
hdfs dfs -put .password /user/$USER/
hdfs dfs -chmod 400 /user/$USER/.password
rm .password

And now when you create a job, you can pass --password-file option and the /user/$USER/.password as the location and it will work.

The fix for flaw #2 I found in this Stack Overflow post.









Tuesday, February 24, 2015

Process 100 files evenly across 10 nodes

I was recently asked to write a script assuming there are 100 files and they're all equal-sized, how to process them evenly across 10 nodes. I went in completely wrong direction suggesting regex where my regex-fu is really not that strong. Thinking it over on a different occasion, I came up with a simple script that will do just that. Of course the simplest solution is the best. As usual, your comments are welcome.


__author__ = 'artem'
nodes = {}
files = []
for filenum in range(100):
filename = 'file_00' + str(filenum)
files.append(filename)
for node in range(10):
subset = files[0:10]
nodes[node] = subset
files[0:10] = []
for node in nodes:
print(node, nodes[node])

and the output:

0 ['file_000', 'file_001', 'file_002', 'file_003', 'file_004', 'file_005', 'file_006', 'file_007', 'file_008', 'file_009']
1 ['file_0010', 'file_0011', 'file_0012', 'file_0013', 'file_0014', 'file_0015', 'file_0016', 'file_0017', 'file_0018', 'file_0019']
2 ['file_0020', 'file_0021', 'file_0022', 'file_0023', 'file_0024', 'file_0025', 'file_0026', 'file_0027', 'file_0028', 'file_0029']
3 ['file_0030', 'file_0031', 'file_0032', 'file_0033', 'file_0034', 'file_0035', 'file_0036', 'file_0037', 'file_0038', 'file_0039']
4 ['file_0040', 'file_0041', 'file_0042', 'file_0043', 'file_0044', 'file_0045', 'file_0046', 'file_0047', 'file_0048', 'file_0049']
5 ['file_0050', 'file_0051', 'file_0052', 'file_0053', 'file_0054', 'file_0055', 'file_0056', 'file_0057', 'file_0058', 'file_0059']
6 ['file_0060', 'file_0061', 'file_0062', 'file_0063', 'file_0064', 'file_0065', 'file_0066', 'file_0067', 'file_0068', 'file_0069']
7 ['file_0070', 'file_0071', 'file_0072', 'file_0073', 'file_0074', 'file_0075', 'file_0076', 'file_0077', 'file_0078', 'file_0079']
8 ['file_0080', 'file_0081', 'file_0082', 'file_0083', 'file_0084', 'file_0085', 'file_0086', 'file_0087', 'file_0088', 'file_0089']
9 ['file_0090', 'file_0091', 'file_0092', 'file_0093', 'file_0094', 'file_0095', 'file_0096', 'file_0097', 'file_0098', 'file_0099']

Process finished with exit code 0

Thursday, February 19, 2015

Unfinished reading list

One great thing about subscription services like Safari Books Online is the choice of all great books available at the moment's notice. What I find happening over and over is that I would start reading a book and never finish. This post is a list of all books I started reading but yet to finish. Here it goes.

ActiveMQ in Action
   excellent book based on the first four chapters I read. I'm not giving up, just shifted in priorities. I intend to finish the book soon. I'm up to chapter 5.

Hadoop: The Definitive Guide 2nd edition 3rd edition 4th edition
   Hadoop bible, a must read for any self-respecting Hadoop engineer. I made a mistake reading this book first, jumping from Microsoft platform to Java and Big Data. I intend to pick it up again when 4th edition is released. This is a really an in-depth book and not recommended for first time users. After having some considerable time with Hadoop, this is the book to turn to. I read the chapters in no particular order.

The Java Programming Language, Fourth Edition
   I was reading this book until I was assigned the AMQ project. I left off at chapter 6. I plan to start reading it again tomorrow.

Hadoop Application Architectures
   Stopped in the middle of chapter 3. I think the reason I stopped was that the book describes some particular use-cases, not necessarily fitting most roles. Covers a wide range of technologies, right now not really on my top priorities list.

Making Java Groovy
   Stopped at chapter 7. I see a lot of potential using Groovy along with Java. However, I am struggling with the choice to continue investing in Groovy vs. giving Scala/Java 8 a chance. I do see a lot of potential using Groovy for unit testing. Setting up test cases in Java can become too wordy. Groovy is here to help. I will continue with the book after some considerable time with Java and HBase.

The Well-Grounded Java Developer: Vital techniques of Java 7 and polyglot programming
   Excellent book but a bit out of scope for me as a recent developer. I am not giving up on the book, but somewhere on the secondary priority list. Stopped at chapter 5.

Using Flume
Started reading this book on a Kindle. A first for me. Couldn't get used to a Kindle and gave up. I do see a lot of potential in using Flume so definitely not giving up there.

This is my list of my unfinished books. I should point out that I try to read all books cover to cover. It is also worth mentioning the last few books I actually did finish.

Learning Chef, Securing Hadoop, Apache Hadoop YARN - Moving beyond MapReduce and Batch Processing.

The next steps are to continue reading the Java book and then the last three most recent HBase books HBase Essentials, Learning HBase and HBase Design Patterns. Perhaps one day I'll post a list of books I'd like to read. Currently the queue is at 60+ books.








Tuesday, February 17, 2015

Book review: Learning Chef

   I am finding myself doing repetitive work once in a while and I've been thinking for a while how to automate the redundant parts of my job. I am familiar with Puppet, Ansible and SaltStack but I never played with Chef before nor Ruby before. I started reading "Learning Chef" and I was surprised to learn that Ruby is the primary language for working with Chef. The other interesting tidbit is that Chef uses Microsoft Powershell for automation on Windows. Onto the review...
   The book is pretty easy to read and is intentionally beginner-friendly. The author right off the bat points out two more professional books for Chef, which I appreciated. Kudos go to the author for following a developer approach, referencing Stack Overflow and Ruby language creator's arguments for better programming practices and argument for scripting language such as Ruby vs. a compiled language like Java for automation.

The 1st chapter is an introduction to automation principles and how Chef came about.

The 2nd chapter walks you through setting up a Chef development environment on Linux, Mac and Windows, step by step. One may think it's a lot of redundant information but keep in mind the target audience.

The 3rd chapter is a brief introduction to Ruby and Chef syntax. As I was reading the last part of the chapter, I realized the true potential of Chef.

The 4th chapter is where it's starting to get interesting. You walk through creating files and deploying them with Chef. As soon as I read the chapter, I was sold! I started getting "light bulb moments" on how to deploy my Java, Python and shell scripts to nodes in a repeatable and controlled fashion.

The 5th chapter introduces you to sand-boxing and previous experience of Vagrant and virtualization will help here. You learn how to deploy your recipes, "instructions" in a sandbox environment to test deployment.

The 6th chapter is where you learn to differentiate between Chef client modes, like solo, local and client modes. You also learn about "ohai" tool, that fetches OS info, as well as how to log messages. This is the first chapter to introduce system administration.

The 7th chapter introduces you to cookbooks and include_recipe statement.

The 8th chapter is all about node attributes and also describes execution precedence, in other words, what takes precedence in order of execution, attribute file, recipe and automatic by ohai tool.

The 9th chapter introduces you to Chef server and I must admit that examples need to be updated. Chef server I presume undergone a name change and I was not able to find the Chef version referenced in the book. After some considerable google-fu and trial and error, I was able to complete the chapter tutorials. Overall, this is the chapter where it finally goes over how to manage a central Chef server and connect nodes to it. If you decide to follow the examples, it is imperative that you complete this tutorial because in chapter ten, the foundation built in this chapter will be used to setup SSL on the slave nodes.

Chapter 10 is broken up into two parts; first part introduces you to chef market place where you can search through multiple recipes shared by the Chef community. The second part builds on chapter ninth' tutorial to setup SSL trust between master and slave.

Chapter 11 covers Chef Zero, which is an in-memory Chef server that you can use to develop recipes.

Chapter 12 is more search functionality.

Chapter 13 is Data Bags and I personally got a lot out of this chapter, this functionality will be pertinent in my work. I enjoyed coverage of encrypted information like passwords the most. It was a bane of mine to manage encryption with homegrown utilities. I see many uses for this going forward. Again I encountered problems with tutorials but as I'm writing this review I am at the end of the tutorials and I find that a lot of my mistakes are from carelessness and I cannot guarantee whether problems were due to my lack of attention or the book. I do however want to mention that some features have changed since the release and some information from commands is in fact different.

Chapter 14 covers roles you can assign to servers, like web servers, database servers.

Chapter 15 covers environments like prod, dev, test, etc. If one should pay attention, it would be this chapter where I personally appreciate that the authors included a full-blown production deployment and a way to manage that vs. development.

Chapter 16 is all about testing, ChefSpec, ServerSpec and FoodCritic. These tools test for specific times in development life cycle and I appreciate the full coverage of scenarios available to users.

Final thoughts

I think the book is beginner friendly but it is easy to get lost in Chef "lingo". I don't think this is my last book on Chef because I still didn't grasp the bulk of it. I noticed that in Appendix A, the author covers Open Source Chef Server and it is basically the same chapter as chapter 9. I realize where I went wrong during my tutorial. I downloaded wrong bits. When I thought I was working with the rpms specific to the chapter, I was actually working with open source Chef server and inadvertently completed the steps in Appendix A.