Saturday, March 19, 2016
Wednesday, March 16, 2016
SequenceIOdocker-ambari
https://github.com/sequenceiq/docker-ambari
cd docker-ambari
sudo make
amb-settings
amb-start-cluster 3
219 sudo route add -net 172.17.0.0/16 docker-machine
cd docker-ambari
sudo make
amb-settings
amb-start-cluster 3
219 sudo route add -net 172.17.0.0/16 docker-machine
Another Vagrant, VB, Ansible HDP setup: from; Henning Kropp
http://henning.kropponline.de/2015/01/11/hdp-ansible-playbook-example/
https://github.com/hkropp/vagrant-hdp
vagrant ssh
[vagrant@one ~]$ cd /vagrant
vagrant@one vagrant]$ ls
ansible bin blueprint.json blueprints conf hostmapping.json inc install_blueprint.sh install.sh manifests master_blueprint.json puppet README.md Vagrantfile
[vagrant@one vagrant]$ ./install_blueprint.sh
In a web browser: http://localhost:8080
username: admin
password: admin
Change the password from the command. See other post.
https://www.blogger.com/blogger.g?blogID=2189538179618563464#editor/target=post;postID=3339040800301665217;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=postname
Go to the Ambari dashboard
YARN / one.hdp CRIT for 2 minutes 1 Connection failed to http://one.hdp:8188
Clck on ops in the top menu bar and watch the installation progress.
https://github.com/hkropp/vagrant-hdp
vagrant ssh
[vagrant@one ~]$ cd /vagrant
vagrant@one vagrant]$ ls
ansible bin blueprint.json blueprints conf hostmapping.json inc install_blueprint.sh install.sh manifests master_blueprint.json puppet README.md Vagrantfile
[vagrant@one vagrant]$ ./install_blueprint.sh
In a web browser: http://localhost:8080
username: admin
password: admin
Change the password from the command. See other post.
https://www.blogger.com/blogger.g?blogID=2189538179618563464#editor/target=post;postID=3339040800301665217;onPublishedMenu=allposts;onClosedMenu=allposts;postNum=0;src=postname
Go to the Ambari dashboard
YARN / one.hdp CRIT for 2 minutes 1 Connection failed to http://one.hdp:8188
Clck on ops in the top menu bar and watch the installation progress.
SequnceIO Hadoop on Docker introduction
Hadoop on Docker introduction
http://blog.sequenceiq.com/blog/2014/04/04/hadoop-docker-introduction/
Create Hadoop clusters in the cloud using a CLI
http://blog.sequenceiq.com/blog/2014/08/07/clodubreak-shell/
Real-time monitoring of Hadoop clusters
http://blog.sequenceiq.com/blog/2014/10/07/hadoop-monitoring/
http://blog.sequenceiq.com/blog/2014/04/04/hadoop-docker-introduction/
Create Hadoop clusters in the cloud using a CLI
http://blog.sequenceiq.com/blog/2014/08/07/clodubreak-shell/
Real-time monitoring of Hadoop clusters
http://blog.sequenceiq.com/blog/2014/10/07/hadoop-monitoring/
Apache Ambari + Spring Shell = Ambari Shell
http://blog.sequenceiq.com/blog/2014/05/26/ambari-shell/
sequenceiq hadoop docker containers
sequenceiq/docker-ambari
https://github.com/sequenceiq/docker-ambari
sequenceiq/cloudbreak
https://github.com/sequenceiq/cloudbreak/tree/master/shell
http://blog.sequenceiq.com/blog/2014/10/30/cloudbreak-devops/
https://github.com/sequenceiq/docker-ambari
sequenceiq/cloudbreak
https://github.com/sequenceiq/cloudbreak/tree/master/shell
http://blog.sequenceiq.com/blog/2014/10/30/cloudbreak-devops/
Docker installation on Linux
Install Docker
https://docs.docker.com/linux/
http://stackoverflow.com/questions/25404597/cannot-connect-to-the-docker-daemon-is-docker-d-running-on-this-host
After running the command below log out and log back in.
Rerun the Docker hello world command. The command should work now.
Rerun the Docker hello world command. The command should work now.
https://docs.docker.com/linux/
$ curl -fsSL https://get.docker.com/ | sh
$ curl -fsSL https://get.docker.com/gpg | sudo apt-key add -
$ docker run hello-world
docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
See 'docker run --help'.
Fix for this problem:http://stackoverflow.com/questions/25404597/cannot-connect-to-the-docker-daemon-is-docker-d-running-on-this-host
After running the command below log out and log back in.
sudo usermod -aG docker
Rerun the Docker hello world command. The command should work now.
docker run hello-world
Rerun the Docker hello world command. The command should work now.
docker run hello-world
Install docker-machine
https://docs.docker.com/machine/
sudo curl -L https://github.com/docker/machine/releases/download/v0.6.0/docker-machine-`uname -s`-`uname -m` > docker-machine
sudo cp docker-machine /usr/local/bin/docker-machine
sudo chmod +x /usr/local/bin/docker-machine
which docker-machine
Install the docker-machine bash completion scripts
https://github.com/docker/machine/tree/master/contrib/completion/bash
git clone git@github.com:docker/machine.git
sudo cp machine/contrib/completion/bash/* /etc/bash_completion.d
https://docs.docker.com/machine/get-started/
(Virtualbox should already be installed)
docker-machine create --driver virtualbox default
docker-machine ls
docker-machine env default
eval "$(docker-machine env default)"
docker run busybox echo hello world
docker-machine ip default
docker run -d -p 8000:80 nginx
curl $(docker-machine ip default):8000
https://docs.docker.com/machine/migrate-to-machine/
boot2docker |
docker-machine |
docker-machine description |
|---|---|---|
| init | create | Creates a new docker host. |
| up | start | Starts a stopped machine. |
| ssh | ssh | Runs a command or interactive ssh session on the machine. |
| save | - | Not applicable. |
| down | stop | Stops a running machine. |
| poweroff | stop | Stops a running machine. |
| reset | restart | Restarts a running machine. |
| config | inspect | Prints machine configuration details. |
| status | ls | Lists all machines and their status. |
| info | inspect | Displays a machine’s details. |
| ip | ip | Displays the machine’s ip address. |
| shellinit | env | Displays shell commands needed to configure your shell to interact with a machine |
| delete | rm | Removes a machine. |
| download | - | Not applicable. |
Tuesday, March 15, 2016
Random notes: Building the build environment with Ansible and Docker
Building the build environment with Ansible and Docker
Marton Sereg09 May 2014
http://blog.sequenceiq.com/blog/2014/05/09/building-the-build-environment-with-ansible-and-docker/
Random notes: Vagrant, Virtualbox, and ansible
https://www.vagrantup.com/docs/
http://docs.ansible.com/ansible/playbooks_intro.html
http://www.ansibletutorials.com/
Ansible courses/tutorials
https://codereviewvideos.com/course/ansible-tutorial/video/looping-in-ansible-with-items
https://metauser.net/2014/07/trying-out-ansible/
https://serversforhackers.com/an-ansible-tutorial
Github: https://github.com/undeadops/ansible-demo
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and
make sure become_method is 'sudo' (default). This feature will be removed in a
future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
http://docs.ansible.com/ansible/become.html
http://stackoverflow.com/questions/29966201/ansible-1-9-1-become-and-sudo-issue
Ansible conditionals
http://stackoverflow.com/questions/32648604/ansible-set-tags-per-item-in-a-loop/32649675
Examples:
https://github.com/henrysher/fedora-infra-ansible/commit/9006a3683242ea9ac92721ea6671f3c59b8e607a
http://docs.ansible.com/ansible/playbooks_loops.html
http://blog.scottlowe.org/2014/09/12/a-quick-introduction-to-vagrant/
http://blog.scottlowe.org/2014/10/22/multi-machine-vagrant-with-yaml/
http://docs.ansible.com/ansible/playbooks_intro.html
http://www.ansibletutorials.com/
Ansible courses/tutorials
https://codereviewvideos.com/course/ansible-tutorial/video/looping-in-ansible-with-items
https://metauser.net/2014/07/trying-out-ansible/
https://serversforhackers.com/an-ansible-tutorial
Github: https://github.com/undeadops/ansible-demo
[DEPRECATION WARNING]: Instead of sudo/sudo_user, use become/become_user and
make sure become_method is 'sudo' (default). This feature will be removed in a
future release. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.
http://docs.ansible.com/ansible/become.html
http://stackoverflow.com/questions/29966201/ansible-1-9-1-become-and-sudo-issue
Ansible conditionals
http://stackoverflow.com/questions/32648604/ansible-set-tags-per-item-in-a-loop/32649675
Examples:
https://github.com/henrysher/fedora-infra-ansible/commit/9006a3683242ea9ac92721ea6671f3c59b8e607a
http://docs.ansible.com/ansible/playbooks_loops.html
http://blog.scottlowe.org/2014/09/12/a-quick-introduction-to-vagrant/
http://blog.scottlowe.org/2014/10/22/multi-machine-vagrant-with-yaml/
Monday, March 14, 2016
Random notes: Building a Hadoop test cluster
WARNING: UNDER CONSTRUCTION
We want to install a Hadoop cluster on either Vagrant or a set of remote Linux servers, copy ssh keys to machines to boot strap Ansible on the servers, install Ansible modules, and Ambari Blueprint. This is a parallel bookkeeping problem. The different con files need the same info -the host names and IP addresses.
https://www.vagrantup.com/docs/multi-machine/index.html
https://gist.github.com/ryanwi/9923791
https://sukhjinderkainth.wordpress.com/2015/08/19/create-a-multi-machine-vagrant-config-file/
# -*- mode: ruby -*-
# vi: set ft=ruby :
# inspired from https://gist.github.com/dlutzy/2469037
# inspired from https://gist.github.com/ryanwi/9923791
boxes = [
{ :name => :hmaster, ip: '192.168.33.10', ssh_port: 2223 },
{ :name => :hslave0, ip: '192.168.33.11', ssh_port: 2224 },
{ :name => :hslave1, ip: '192.168.33.12', ssh_port: 2225 },
{ :name => :hslave2, ip: '192.168.33.13', ssh_port: 2226 },
{ :name => :hslave3, ip: '192.168.33.14', ssh_port: 2227 },
{ :name => :hslave4, ip: '192.168.33.15', ssh_port: 2228 },
]
CUSTOM_CONFIG = {
"BOX_NAME" => "ubuntu/trusty64",
"BOX_URL" => "https://vagrantcloud.com/ubuntu/boxes/trusty64",
"HEADLESS" => false
}
Vagrant.configure("2") do |config|
# headless? uncomment this to have the VM's window available
config.vm.provider :virtualbox do |vb|
vb.gui = CUSTOM_CONFIG['HEADLESS']
end
# Disable default ssh in order to manually assign to a known value
# https://github.com/mitchellh/vagrant/issues/3232
config.vm.network :forwarded_port, guest: 22, host: 2222, id: "ssh", disabled: true
boxes.each do |opts|
config.vm.define opts[:name] do |boxconfig|
boxconfig.vm.box = CUSTOM_CONFIG['BOX_NAME']
boxconfig.vm.box_url = CUSTOM_CONFIG['BOX_URL']
boxconfig.vm.hostname = "%s.vagrant" % opts[:name].to_s
boxconfig.vm.network "private_network", ip: opts[:ip]
boxconfig.vm.network :forwarded_port, guest: 22, host: opts[:ssh_port]
end
end
# provisioning with ansible
# config.vm.provision :ansible do |ansible|
# ansible.playbook = "./provisioning/site.yml"
# end
end
$ vagrant status
Current machine states:
hmaster running (virtualbox)
hslave0 running (virtualbox)
hslave1 running (virtualbox)
hslave2 running (virtualbox)
hslave3 running (virtualbox)
hslave4 running (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
Now lets add an include file with the server vars to the above Vagrantfile
http://stackoverflow.com/questions/16708917/how-do-i-include-variables-in-my-vagrantfile
Ambari Ansible installation.
http://dataminded.be/blog/how-install-hortonworks-hdp-22-ubuntu-1204
Hortonworks blueprints
https://medium.com/@b23llc/managing-distributed-data-products-with-ansible-and-ambari-44c7175555d8#.ytz5ceqsn
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step0:PrepareAmbariServerandAgents
Blueprint Details
Generate the ssh key for the Hadoop servers
ssh-keygen
Copy the id_rsa.pub key to the Vagrant folder
cp id_rsa.pub /Vagrant
Copy the ssh id_rsa.pub key to the Vagrant machines
We want to install a Hadoop cluster on either Vagrant or a set of remote Linux servers, copy ssh keys to machines to boot strap Ansible on the servers, install Ansible modules, and Ambari Blueprint. This is a parallel bookkeeping problem. The different con files need the same info -the host names and IP addresses.
https://www.vagrantup.com/docs/multi-machine/index.html
https://gist.github.com/ryanwi/9923791
https://sukhjinderkainth.wordpress.com/2015/08/19/create-a-multi-machine-vagrant-config-file/
# -*- mode: ruby -*-
# vi: set ft=ruby :
# inspired from https://gist.github.com/dlutzy/2469037
# inspired from https://gist.github.com/ryanwi/9923791
boxes = [
{ :name => :hmaster, ip: '192.168.33.10', ssh_port: 2223 },
{ :name => :hslave0, ip: '192.168.33.11', ssh_port: 2224 },
{ :name => :hslave1, ip: '192.168.33.12', ssh_port: 2225 },
{ :name => :hslave2, ip: '192.168.33.13', ssh_port: 2226 },
{ :name => :hslave3, ip: '192.168.33.14', ssh_port: 2227 },
{ :name => :hslave4, ip: '192.168.33.15', ssh_port: 2228 },
]
CUSTOM_CONFIG = {
"BOX_NAME" => "ubuntu/trusty64",
"BOX_URL" => "https://vagrantcloud.com/ubuntu/boxes/trusty64",
"HEADLESS" => false
}
Vagrant.configure("2") do |config|
# headless? uncomment this to have the VM's window available
config.vm.provider :virtualbox do |vb|
vb.gui = CUSTOM_CONFIG['HEADLESS']
end
# Disable default ssh in order to manually assign to a known value
# https://github.com/mitchellh/vagrant/issues/3232
config.vm.network :forwarded_port, guest: 22, host: 2222, id: "ssh", disabled: true
boxes.each do |opts|
config.vm.define opts[:name] do |boxconfig|
boxconfig.vm.box = CUSTOM_CONFIG['BOX_NAME']
boxconfig.vm.box_url = CUSTOM_CONFIG['BOX_URL']
boxconfig.vm.hostname = "%s.vagrant" % opts[:name].to_s
boxconfig.vm.network "private_network", ip: opts[:ip]
boxconfig.vm.network :forwarded_port, guest: 22, host: opts[:ssh_port]
end
end
# provisioning with ansible
# config.vm.provision :ansible do |ansible|
# ansible.playbook = "./provisioning/site.yml"
# end
end
$ vagrant status
Current machine states:
hmaster running (virtualbox)
hslave0 running (virtualbox)
hslave1 running (virtualbox)
hslave2 running (virtualbox)
hslave3 running (virtualbox)
hslave4 running (virtualbox)
This environment represents multiple VMs. The VMs are all listed
above with their current state. For more information about a specific
VM, run `vagrant status NAME`.
Now lets add an include file with the server vars to the above Vagrantfile
http://stackoverflow.com/questions/16708917/how-do-i-include-variables-in-my-vagrantfile
Ambari Ansible installation.
http://dataminded.be/blog/how-install-hortonworks-hdp-22-ubuntu-1204
Hortonworks blueprints
https://medium.com/@b23llc/managing-distributed-data-products-with-ansible-and-ambari-44c7175555d8#.ytz5ceqsn
https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-Step0:PrepareAmbariServerandAgents
Blueprint Details
Manually Prepare Ambari Server and Agents
- Perform your Ambari Server install and setup.
yum install ambari-serverambari-server setup - After setup completes, start your Ambari Server.
ambari-server start - Install Ambari Agents on all of the hosts you plan to include in your cluster.
yum install ambari-agent - Set the Ambari Server on the Ambari Agents.
vi /etc/ambari-agent/conf/ambari-agent.ini - Set hostname= to the Fully Qualified Domain Name for the Ambari Server. Save and exit.
hostname=c6401.ambari.apache.org - Start the Agents to initiate registration to Server.
ambari-agent start - Confirm the Agent hosts are registered with the Server.
http://your.ambari.server:8080/api/v1/hosts
Generate the ssh key for the Hadoop servers
ssh-keygen
Copy the id_rsa.pub key to the Vagrant folder
cp id_rsa.pub /Vagrant
Script
cat /Vagrant/id_rsa.pub >> `/.ssh/authorized_keys
Copy the ssh id_rsa.pub key to the Vagrant machines
ssh root@MachineB 'bash -s' < local_script.…
Tuesday, March 8, 2016
digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-an-ubuntu-12-04-vps
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-an-ubuntu-12-04-vps
Random notes: SSH tricks for Vagrant and Ambari
Follow the instructions on the following link-see the note below on logging in as root:
https://ambari.apache.org/1.2.2/installing-hadoop-using-ambari/content/ambari-chap1-5-2.html
https://ambari.apache.org/1.2.2/installing-hadoop-using-ambari/content/ambari-chap1-5-2.html
To login as root on the virtual machine:
vagrant up
vagrant ssh
sudo - root
(password vagrant-change it!!!!)
From another terminal in the Vagrant directory that you launched the VM test the connection to the vagrant VM. Vagrant installs the .ssh/id_rsa.pub to the VM
ssh vagrant@192.168.33.100
https://serversforhackers.com/ssh-tricks/
http://www.cyberciti.biz/faq/linux-unix-applesox-ssh-password-on-command-line/
http://askubuntu.com/questions/46424/adding-ssh-keys-to-authorized-keys
An alternative way to install your public key in the remote machine's
authorized_keys:
http://www.cyberciti.biz/tips/linux-multiple-ssh-key-based-authentication.html
https://gist.github.com/jexchan/2351996/
http://www.debian-administration.org/article/152/Password-less_logins_with_OpenSSH
https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html
https://www.racf.bnl.gov/docs/authentication/ssh/sshkeygenunix
https://ambari.apache.org/1.2.2/installing-hadoop-using-ambari/content/ambari-chap1-5-2.html
https://ambari.apache.org/1.2.2/installing-hadoop-using-ambari/content/ambari-chap1-5-2.html
To login as root on the virtual machine:
vagrant up
vagrant ssh
sudo - root
(password vagrant-change it!!!!)
From another terminal in the Vagrant directory that you launched the VM test the connection to the vagrant VM. Vagrant installs the .ssh/id_rsa.pub to the VM
ssh vagrant@192.168.33.100
https://serversforhackers.com/ssh-tricks/
http://www.cyberciti.biz/faq/linux-unix-applesox-ssh-password-on-command-line/
http://askubuntu.com/questions/46424/adding-ssh-keys-to-authorized-keys
An alternative way to install your public key in the remote machine's
authorized_keys:
cat ~/.ssh/id_rsa.pub | ssh root@HOST "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys
Restart sshd on the remote host
sudo /etc/init.d/sshd restart
Stopping sshd: [ OK ]
Starting sshd: [ OK ]
Now test the passwordless ssh login to root@host
http://www.cyberciti.biz/tips/linux-multiple-ssh-key-based-authentication.html
https://gist.github.com/jexchan/2351996/
http://www.debian-administration.org/article/152/Password-less_logins_with_OpenSSH
https://www.vagrantup.com/docs/vagrantfile/ssh_settings.html
https://www.racf.bnl.gov/docs/authentication/ssh/sshkeygenunix
Monday, March 7, 2016
Random notes: Installing Ansible in a virtual Python environment on Mac OS X and running Ansible in a VirtualBox and Vagrant env
Download and install Vagrant
https://www.vagrantup.com/downloads.html
Read the docs!
https://www.vagrantup.com/docs/
Download and install VirtualBox
Read the docs!
https://www.vagrantup.com/docs/
Download and install VirtualBox
http://docs.python-guide.org/en/latest/dev/virtualenvs/
mkdir ansible_ve
cd ansible_ve
virtualenv venv
source venv/bin/activate
sudo easy_install ansible
When you are done working in the virtual environment for the moment, you can deactivate it:
deactivate
UNDER CONSTRUCTION
UNDER CONSTRUCTION
Thursday, March 3, 2016
Random notes: Linux Mint installation + virtual box, vagrant, Ambari Blueprints, Ansible...
https://www.linuxmint.com/download.php
Use md5
https://www.reddit.com/r/sysadmin/comments/2fmkvq/vagrat_saltstack_ansible_docker_chef_puppet/
A reddit about some of the tools in the rest of this post.
https://www.upguard.com/articles/docker-vs-vagrant
https://www.virtualbox.org/wiki/Linux_Downloads
Use the instructions for: Debian-based Linux distributions
Use sha256sum
Test the virtual box setup by running virtual box from the Linux applications menu
-if you get the vt-x not enabled in the BIOS message
--restart the computer, go into the bios and change the virtual support configuration from disabled to enabled.
Test the virtual box setup by running virtual box from the Linux applications menu
-if you get the vt-x not enabled in the BIOS message
--restart the computer, go into the bios and change the virtual support configuration from disabled to enabled.
SSH
https://serversforhackers.com/ssh-tricks/
scroll down...
ssh -p 2222 vagrant@vagrant-centos65.vagrantup.com
Last login: Tue Mar 8 17:44:27 2016 from 10.0.2.2
ping 10.0.2.15
PING 10.0.2.15 (10.0.2.15) 56(84) bytes of data.
^C
--- 10.0.2.15 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 3999ms
http://docs-v1.vagrantup.com/v1/docs/host_only_networking.html
http://superuser.com/questions/570261/cant-ping-to-vagrant-box
https://www.vagrantup.com/docs/getting-started/networking.html
http://stackoverflow.com/questions/23074412/how-to-set-host-key-checking-false-in-ansible-inventory-file
http://superuser.com/questions/570261/cant-ping-to-vagrant-box
Download and verify the sha256 sum.
Then click on the link and Mint will install this using the gedi package installer.
https://programmaticponderings.wordpress.com/2014/12/14/installing-puppet-master-and-agents-on-multiple-vm-using-vagrant-and-virtualbox/
http://www.vagrantbox.es
http://leonidmirsky.com/ansible/hadoop/devops/2013/11/19/creating-hadoop-test-environment-with-ansible-and-vagrant.html
Try out a virtual hadoop config.
https://dantehranian.wordpress.com/2015/01/20/ansible-vs-puppet-hands-on-with-ansible/
https://www.ansible.com
Tutorials
https://github.com/leucos/ansible-tuto/tree/master/step-01#inventory
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-an-ubuntu-12-04-vps
https://www.ansible.com/videos
Watch the quick start video
Playbook use YAML
http://yaml.org
http://docs.ansible.com/ansible/YAMLSyntax.html
Simple ansible command to ping all servers
ansible -m ping -u vagrant all
ansible-playbook.yaml -e "username:vagrant"
command: does not use the shell
shell: uses the shell
http://docs.ansible.com/ansible/playbooks_intro.html
http://jinja.pocoo.org/docs/dev/
http://ryandlane.com/blog/2014/08/04/moving-away-from-puppet-saltstack-or-ansible/
https://galaxy.ansible.com/list#/roles?page=1&page_size=10&autocomplete=hadoop
https://programmaticponderings.wordpress.com/2014/12/14/installing-puppet-master-and-agents-on-multiple-vm-using-vagrant-and-virtualbox/
http://www.vagrantbox.es
Vagrantbox.es
Vagrant is an amazing tool for managing virtual machines via a simple to use command line interface. With a simple
vagrant up you can be working in a clean environment based on a standard template.
These standard templates are called base boxes, and this website is simply a list of boxes people have been nice enough to make publicly available.
http://superuser.com/questions/671191/how-to-ssh-between-a-cluster-of-vagrant-guest-vms?rq=1http://leonidmirsky.com/ansible/hadoop/devops/2013/11/19/creating-hadoop-test-environment-with-ansible-and-vagrant.html
Try out a virtual hadoop config.
https://dantehranian.wordpress.com/2015/01/20/ansible-vs-puppet-hands-on-with-ansible/
https://www.ansible.com
Tutorials
https://github.com/leucos/ansible-tuto/tree/master/step-01#inventory
https://www.digitalocean.com/community/tutorials/how-to-install-and-configure-ansible-on-an-ubuntu-12-04-vps
https://www.ansible.com/videos
Watch the quick start video
Playbook use YAML
http://yaml.org
http://docs.ansible.com/ansible/YAMLSyntax.html
Simple ansible command to ping all servers
ansible -m ping -u vagrant all
ansible-playbook
command: does not use the shell
shell: uses the shell
http://docs.ansible.com/ansible/playbooks_intro.html
http://jinja.pocoo.org/docs/dev/
http://ryandlane.com/blog/2014/08/04/moving-away-from-puppet-saltstack-or-ansible/
https://galaxy.ansible.com/list#/roles?page=1&page_size=10&autocomplete=hadoop
Wednesday, March 2, 2016
How to create a live bootable USB on a Mac to install Linux on a PC
From http://www.webupd8.org/2009/04/4-ways-to-create-bootable-live-usb.html
Using Diskutil (Mac OS X only)
- Download the desired .img file
- Open a Terminal (under Utilities)
- Run diskutil list to get the current list of devices
- Insert your flash media
- Run diskutil list again and determine the device node assigned to your flash media (e.g. /dev/disk2)
- Run diskutil unmountDisk /dev/diskN
- Execute sudo dd if=/path/to/downloaded.img of=/dev/diskN bs=1m
- Run diskutil eject /dev/diskN and remove your flash media when the command completes
- Insert the USB into the PC
- Enter the BIOS setup in your PC
- Set the USB to boot first
- Reboot the computer
- Install Linux
Friday, September 11, 2015
Newbie Scala build intro
1. Install Java 8
If you forget to do this you will probably see all sorts of unexplained bugs.
2. Download the Intellij IDE
https://www.jetbrains.com/idea/download/
or the scala-ide based on Eclipse
http://scala-ide.org
Check out the plugins for Intellij
https://www.jetbrains.com/idea/plugins/
Install the JetBrains official Scala plugin
3. Create a directory structure (optionally first create your revision control repo structure and cd to it e.g. git...) using this handy script which has some optional sections commented out.
http://alvinalexander.com/sbtmkdirs
http://grosdim.blogspot.com/2013/01/quick-sbt-tutorial.html
http://www.scala-sbt.org/0.13/tutorial/index.html
4. Optionally install sbteclipse so you can generate an eclipse project if you use scala-ide
https://github.com/typesafehub/sbteclipse/
https://github.com/typesafehub/sbteclipse/wiki/Installing-sbteclipse
If you forget to do this you will probably see all sorts of unexplained bugs.
2. Download the Intellij IDE
https://www.jetbrains.com/idea/download/
or the scala-ide based on Eclipse
http://scala-ide.org
Check out the plugins for Intellij
https://www.jetbrains.com/idea/plugins/
Install the JetBrains official Scala plugin
3. Create a directory structure (optionally first create your revision control repo structure and cd to it e.g. git...) using this handy script which has some optional sections commented out.
http://alvinalexander.com/sbtmkdirs
http://grosdim.blogspot.com/2013/01/quick-sbt-tutorial.html
http://www.scala-sbt.org/0.13/tutorial/index.html
4. Optionally install sbteclipse so you can generate an eclipse project if you use scala-ide
https://github.com/typesafehub/sbteclipse/
https://github.com/typesafehub/sbteclipse/wiki/Installing-sbteclipse
emacs ~/.sbt/0.13/plugins/plugins.sbt
(~/.sbt/0.13/plugins probably does not exist and either use emacs to make it or mkdir ~/.sbt/0.13/plugins and then use your favorite editor)
add
addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.5.0")
"After adding the sbteclipse plugin like this, you should either start sbt or, if it was already started, reload the current session by executing the reload command. If everything worked, you should have the new command eclipse available. Now read on about how to use sbteclipse."
$ sbt
Use of ~/.sbtconfig is deprecated, please migrate global settings to /usr/local/etc/sbtopts
[info] Loading global plugins from /Users/depappas/.sbt/0.13/plugins
[info] Updating {file:/Users/depappas/.sbt/0.13/plugins/}global-plugins...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading https://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbteclipse/sbteclipse-plugin/scala_2.10/sbt_0.13/2.5.0/jars/sbteclipse-plugin.jar ...
[info] [SUCCESSFUL ] com.typesafe.sbteclipse#sbteclipse-plugin;2.5.0!sbteclipse-plugin.jar (1808ms)
[info] downloading https://repo.typesafe.com/typesafe/ivy-releases/com.typesafe.sbteclipse/sbteclipse-core/scala_2.10/sbt_0.13/2.5.0/jars/sbteclipse-core.jar ...
[info] [SUCCESSFUL ] com.typesafe.sbteclipse#sbteclipse-core;2.5.0!sbteclipse-core.jar (2512ms)
[info] Done updating.
[info] Set current project to html_product_website_generator (in build file:/Users/depappas/scala_test/html_product_website_generator/)
> eclipse
[info] About to create Eclipse project files for your project(s).
[info] Updating {file:/Users/depappas/scala_test/html_product_website_generator/}html_product_website_generator...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Successfully created Eclipse project files for project(s):
[info] html_product_website_generator
>
5. Configure scala-ide
http://scala-ide.org/docs/user/gettingstarted.html
5. Open up the set project in Intellij
File->Open
select the project
Or in Eclipse
Once you have installed and generated the Eclipse project files using one of the above plug-ins, start Eclipse. Use
File → Import → General/Existing Project into Workspace. Select the directory containing your project as root directory, select the project and hit Finish. And Voila.
6. If you want to use scala.xml and get compile errors try this:
Add this line to the build.sbt file
libraryDependencies += "org.scala-lang.modules" % "scala-xml_2.11" % "1.0.5"
$ sbt
> update
> eclipse
Go to Eclipse and refresh the project
The compile errors in Eclipse should be gone.
If you get this error:
UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scala-lang#scala-library;2.11.7 : not found
Then find the right scala.xml version:
Lets find the right version of the scala.xml to add to the build.sb file.
Now lets add it to the build.sbt file.
Lets check in the cache dir: ~/.ivy2/cache
for the missing libs
Thursday, September 10, 2015
Upgrading to the Java 8 SDK on a Mac
Things seem to be getting simpler...
Download and install the dmg from Oracle
Read the post installation docs
Update the system to use Java 8
Download and install the dmg from Oracle
- http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
Read the post installation docs
- http://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jdk.html#CHDBADCG
Update the system to use Java 8
$ /usr/libexec/java_home -v 1.8.0_60 --exec javac -version
javac 1.8.0_60
$ which java
/usr/bin/java
Friday, July 31, 2015
Using emacs to paste multiple lines into a rectangle-this does not seem to work in Eclipse block mode
Lets say you have several lines that you want to paste into a rectangle so that the lines are inserted into each block.
Copy the code to paste into the empty braces,
case ACCESSORIES : { }
case AVAILABILITY : { }
case BRANDLOGO : { }
case BRANDNAME : { }
case BRANDPARTNUMBER : { }
case BRANDPRODUCTPAGE : { }
case BRANDSUGGESTEDPRICE : { }
case BRANDSUPPORTPHONE : { }
case BRANDWEBSITE : { }
Availability schema = SchemaRegistry.getSchemaByClass(Availability.class);
String schemaName = schema.getName();
return schemaName;
Now you want to paste in the following code into each set of braces.
This paste operation does not seem to work in Eclipse. However, in Emacs use insert rectangle (cantle x-r-t) to paste the lines into each set of braces. Use control-space and the arrow keys to select the rectangle inside of the braces. Then use control-Y or command-v on the Mac to paste the contents into the mini buffer and hit return. the multiple lines of text are pasted into each set of braces
case ACCESSORIES : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
case AVAILABILITY : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
case BRANDLOGO : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
case BRANDNAME : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
case BRANDPARTNUMBER : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
case BRANDPRODUCTPAGE : {
Accessories schema = SchemaRegistry.getSchemaByClass(Accessories.class);
String schemaName = schema.getName();
return schemaName;
}
}
Saturday, February 1, 2014
Python3 set operations
how-to-subtraction-two-list-in-python
Python has a built-in datatype for an unordered collection of (hashable) things, called a
EDIT: @mdwhatcott points out that you want to check for duplicates.
best-way-to-check-if-a-list-is-empty
set. If you convert both lists to sets, the comparison will be unordered.set(x) == set(y)
Documentation on setEDIT: @mdwhatcott points out that you want to check for duplicates.
set
ignores these, so you need a similar data structure that also keeps
track of the number of items in each list. This is called a multiset; the best approximation in the standard library is a collections.Counter:>>> import collections
>>> compare = lambda x, y: collections.Counter(x) == collections.Counter(y)
>>>
>>> compare([1,2,3], [1,2,3,3])
False
>>> compare([1,2,3], [1,2,3])
True
>>> compare([1,2,3,3], [1,2,2,3])
False
>>>
best-way-to-check-if-a-list-is-empty
if not a:
print "List is empty"
Using the implicit booleanness of the empty list is quite pythonic.
Python3 command line help
use help(dict) on the Python command line and replace dict with the troublesome object to see a list of its helper methods etc., e.g. items(...)
D.items() -> list of D's (key, value) pairs, as 2-tuples
–
>>> help(list)
Help on class list in module builtins:
class list(object)
| list() -> new empty list
| list(iterable) -> new list initialized from iterable's items
|
| Methods defined here:
|
| __add__(...)
| x.__add__(y) <==> x+y
|
| __contains__(...)
| x.__contains__(y) <==> y in x
|
| __delitem__(...)
| x.__delitem__(y) <==> del x[y]
|
| __eq__(...)
| x.__eq__(y) <==> x==y
|
| __ge__(...)
| x.__ge__(y) <==> x>=y
|
| __getattribute__(...)
| x.__getattribute__('name') <==> x.name
|
| __getitem__(...)
| x.__getitem__(y) <==> x[y]
|
| __gt__(...)
| x.__gt__(y) <==> x>y
|
| __iadd__(...)
| x.__iadd__(y) <==> x+=y
|
| __imul__(...)
| x.__imul__(y) <==> x*=y
|
| __init__(...)
| x.__init__(...) initializes x; see help(type(x)) for signature
|
| __iter__(...)
| x.__iter__() <==> iter(x)
|
| __le__(...)
| x.__le__(y) <==> x<=y
|
| __len__(...)
| x.__len__() <==> len(x)
|
:
>>> help(list)
Help on class list in module builtins:
class list(object)
| list() -> new empty list
| list(iterable) -> new list initialized from iterable's items
|
| Methods defined here:
|
| __add__(...)
| x.__add__(y) <==> x+y
|
| __contains__(...)
| x.__contains__(y) <==> y in x
|
| __delitem__(...)
| x.__delitem__(y) <==> del x[y]
|
| __eq__(...)
| x.__eq__(y) <==> x==y
|
| __ge__(...)
| x.__ge__(y) <==> x>=y
|
| __getattribute__(...)
| x.__getattribute__('name') <==> x.name
|
| __getitem__(...)
| x.__getitem__(y) <==> x[y]
|
| __gt__(...)
| x.__gt__(y) <==> x>y
|
| __iadd__(...)
| x.__iadd__(y) <==> x+=y
|
| __imul__(...)
| x.__imul__(y) <==> x*=y
|
| __init__(...)
| x.__init__(...) initializes x; see help(type(x)) for signature
|
| __iter__(...)
| x.__iter__() <==> iter(x)
|
| __le__(...)
| x.__le__(y) <==> x<=y
|
| __len__(...)
| x.__len__() <==> len(x)
|
:
Python3 list operations
how-to-subtraction-two-list-in-python
>>> List1=[3,5,6]
>>> List2=[3,7,2]
>>> [x1 - x2 for (x1, x2) in zip(List1, List2)]
[0, -2, 4]
Python sets
>>> a = [1,2,3,4]
>>> b = [3,4,5,6]
>>> set(a)
{1, 2, 3, 4}
>>> (set(b)
{3, 4, 5, 6}
>>> sa= set(a)
>>> sb= set(b)
>>> sa-sb
{1, 2}
>>> sb-sa
{5, 6}
>>> sa.intersection(sb)
{3, 4}
>>> sa.intersection(sb)
{3, 4}
>>> 3 in sa
True
>>> 10 in sa
False
>>> len(sa)
4
>>> sa.issubset(sb)
False
>>> sa.union(sb)
{1, 2, 3, 4, 5, 6}
>>> sa.difference(sb)
{1, 2}
>>> sb.difference(sa)
{5, 6}
>>> set(a) - set(b)
{1, 2}
>>> set(b) - set(a)
{5, 6}
>>> set(a) - set(a)
set()
>>> sa.symmetric_difference(sb)
{1, 2, 5, 6}
>>> sb.symmetric_difference(sa)
{1, 2, 5, 6}
>>> List1=[3,5,6]
>>> List2=[3,7,2]
>>> [x1 - x2 for (x1, x2) in zip(List1, List2)]
[0, -2, 4]
Python sets
>>> a = [1,2,3,4]
>>> b = [3,4,5,6]
>>> set(a)
{1, 2, 3, 4}
>>> (set(b)
{3, 4, 5, 6}
>>> sa= set(a)
>>> sb= set(b)
>>> sa-sb
{1, 2}
>>> sb-sa
{5, 6}
>>> sa.intersection(sb)
{3, 4}
>>> sa.intersection(sb)
{3, 4}
>>> 3 in sa
True
>>> 10 in sa
False
>>> len(sa)
4
>>> sa.issubset(sb)
False
>>> sa.union(sb)
{1, 2, 3, 4, 5, 6}
>>> sa.difference(sb)
{1, 2}
>>> sb.difference(sa)
{5, 6}
>>> set(a) - set(b)
{1, 2}
>>> set(b) - set(a)
{5, 6}
>>> set(a) - set(a)
set()
>>> sa.symmetric_difference(sb)
{1, 2, 5, 6}
>>> sb.symmetric_difference(sa)
{1, 2, 5, 6}
python: using dictionaries instead of case statements
Switch Statements in Python
Replace with
maps-as-switch.py
response, data = somefunc()
if response == "this":
do_this_with(data)
elif response == "that":
do_that_with(data)
elif response == "huh":
duh(data)
# lots more elifs.
else:
prevent_horrible_crash(data)
Replace with
maps-as-switch.py
|
Subscribe to:
Posts (Atom)

