Thursday, March 5, 2015

Running saved sqoop 1.4.4 jobs with password file

This has been bugging me for a long time. I've been trying to pass password to sqoop job securely for the longest time with no avail. Apparently, my approach was flawed by two things. First of all, the sqoop user guide 1.4.4 has incorrect spelling for password file option in the example. The documentation has the correct option but if you're like me and follow the example you will get it wrong.

wrong way in Sqoop 1.4.4 user guide:
sqoop import --connect jdbc:mysql://database.example.com/employees \
    --username venkatesh --passwordFile ${user.home}/.password

right way in Sqoop 1.4.5 user guide:
sqoop import --connect jdbc:mysql://database.example.com/employees \
    --username venkatesh --password-file ${user.home}/.password

the other flaw is that when you create the password file with "echo" command, make sure you don't append a new line character to the password. Here's a handy script to create a password:

echo -n "password" > .password
hdfs dfs -put .password /user/$USER/
hdfs dfs -chmod 400 /user/$USER/.password
rm .password

And now when you create a job, you can pass --password-file option and the /user/$USER/.password as the location and it will work.

The fix for flaw #2 I found in this Stack Overflow post.









Post a Comment