Install VirtualBox and Vagrant
Download 1.1.0 release
wget http://www.apache.org/dist/bigtop/bigtop-1.1.0/bigtop-1.1.0-project.tar.gz
uncompress the tarball
tar -xvzf bigtop-1.1.0-project.tar.gz
change directory to bigtop-1.1.0/bigtop-deploy/vm/vagrant-puppet-vm
cd bigtop-1.1.0/bigtop-deploy/vm/vagrant-puppet-vm
here you can review the README but to keep it short you can edit the vagrantconfig.yaml for any additional customization like changing VM memory, OS, number of CPUs, components (e.g. hadoop, spark, tez, hama, solr) etc and also number of VMs you'd like to provision. This last part is the killer feature, you can provision a Sandbox with multiple nodes, not a single VM. Same is true with Docker provisioner but I can't confirm that for you. Feel free to read the README in bigtop-1.1.0/bigtop-deploy/vm/vagrant-puppet-docker for that approach.
then you can start provisioning your custom sandbox with
vagrant up
wait 5-10min and then you can use standard Vagrant commands to interact with your custom Sandbox.
vagrant ssh bigtop1
now just create your local user and off you go
sudo -u hdfs hdfs dfs -mkdir /user/vagrant
sudo -u hdfs hdfs dfs -chown -R vagrant:hdfs /user/vagrant
for your convenience, add the bigtop machine(s) to /etc/hosts
Now, you're probably wondering why would I use Bigtop over regular sandbox? Well, Sandbox has been getting pretty resource heavy and has a lot of components. I like to provision a small cluster with just a few components like hadoop, spark, yarn and pig. Bigtop makes this possible and runs easily within a memory strapped VM. One downside is that with the latest release, Spark is at 1.5.0 and Hortonworks Sandbox is at 1.6.0, story is the same with other components. There are version gaps and if you can look past it, you have a quick way to prototype without much fuss! This is by no means meant to steal thunder from an excellent Ambari quick start guide, this is meant to demonstrate yet another approach from a rich ecosystem of Hadoop tools.
No comments:
Post a Comment