Adding WASB blob as HDFS replacement in Hortonworks HDP 2.3.2

DISCLAIMER: it was tested on HDP 2.3.2 only. There are two blocking JIRAs preventing usage of blob storage as primary filesystem on HDP 2.3.0. For HBase, you need to use page blob instead of block blob.

First things first, install Azure CLI for Mac or use Azure portal. The steps below are for CLI.

azure login
enter username
enter password
azure storage account create storageaccountname --type LRS


azure storage account keys list storageaccountname

note the account keys, you will need them in the next step

azure storage container create storagecontainername --account-name storageaccountname --account-key accountkeystring

just to validate it was created

azure storage blob list storagecontainernae --account-name storageaccountname --account-key 

Once the previous steps have been completed, go to Ambari UI and edit the core-site.xml


In addition to these properties, you need to replace fs.defaultFS property with the wasb path.


These properties and their descriptions are discussed in hadoop-azure documentation. If you choose to install HBase you also need to edit hbase-site.xml and modify hbase.rootDir property.

Now restart the cluster for changes to take effect and start using the cluster. For HBase, there are some open JIRAs and your usage may vary. I encountered the following error when I tried to pre-split and drop/create the same table over and over. The fix is coming in Hadoop 2.8 so until then, beware of acquired lease messages on HBase.

Comments

Post a Comment

Popular posts from this blog

Vista Vulnerability Report mentions Ubuntu 6.06 LTS

Update: I posted this on MIXX since DIGG was taken care of by someone already, except that that user didn't really write a good preview for it so it got buried. This report is generating some buzz though. I emailed this report to Mark Shuttleworth of Ubuntu fame and little did I know, HE ANSWERED!!!! Whoever answered me on that side of the e-mail pipe liked the report and was wondering if it was official and whether he could quote from it. It definitely is official and I forwarded him the e-mail with this report, he said he will pass it down internally. That made my day, it's amazing that a man of his statute, that's if it was him, answers e-mails from regular people like me. That's what I call a real genuine pioneer! Thank you Mark and all of the Ubuntu Staff. Read
Read more

Running CockroachDB with Docker Compose and Minio, Part 2

This is my second post on creating a multi-service architecture with docker-compose. This is meant to be a learning exercise and typically docker-compose is used to set up a local development environment rather than a production-ready set up. I regularly, find myself building these environments to reproduce customer bugs. For a production-specific application, refer to your platform vendor documentation. At some later time, I will cover Kubernetes deployments that can be used as a stepping stone for a real-world application. Until then, let's focus on the task at hand. We're building a microservice architecture with CockroachDB writing changes in real-time to an S3 bucket in JSON format. S3 bucket is served by a service called Minio. It can act like an S3 appliance on premise or serve as a local gateway to your cloud storage. Let's dig in:
Read more

Doing "print screen" on a Mac is a pain in the ass

I am catching up with blogging today and I need to take a screenshot of my screen. I have a Macbook Pro with Ubuntu in a virtual machine. There is no printscreen button and a quick search returned "Apple + Shift + 3". Mac fanboys think that's easy and better than PC? So here's a problem. If you want to take a print screen in VM, that key combination doesn't work. What do I do besides opening the screenshot application and doing it half-assed way? I am sure you can edit key bindings but that should not be that way. If you know of any other ways, let me know.
Read more