Automating the generation of Vagrant and Ansible host/server/node config files

Check back later. We will release a set of scripts for bringing up either a vagrant system or dedicated servers running Hortonworks HDP on Centos 7. We will upload the setup scripts and Ansible files and Vagrantfile to github.

Under construction as we bring up a new devops environment for deploying our HDP cluster.
-This post will be updated as we add the scripts to do the conversions.
-We have a boot strapping script for setting up the keyless ssh which we have added to git repo. After everything is debugged we will release the git repo.
Our sysadmin/devops engineer quit and we (the software developers) decided to learn devops. Hence the messy series of posts on Ansible that appeared prior to this post. Trying to clean the blog up...

The aim of this approach is to eliminate the parallel bookkeeping required to copy/convert host information into different file formats and to have a flow which works in Vagrant and on new dedicated servers.

Status: 
We released the Vagrant bring up script using the servers.json file.
We released the ssh key copying program that uses the servers.json file.

Current: 
Next: Ansible bring up

Goal: 

1. Build/update automatically for Vagrant or Ansible using a single server config file that generates all other files.
2. Build the same configuration on either a Vagrant cluster or a dedicated cluster, with the only difference being the number of HDP workers.

Why: Avoid consistency problems and parallel bookkeeping for host/server information.

Flow:

Preparation: Create the host_config.json file with the host names, ip address, optional user names, passwords, configuration name, ...

servers.json 

{
    "configurationId": "Hetzner HDP Cluster",
    "servers": [
 {
     "name": "hmaster",
     "box": "centos/7",
     "ram": 512,
     "vcpu": 1,
     "ip_addr": "192.168.100.101",
            "rootPassWord": "abc123456789"
 },
 {
     "name": "hslave0",
     "box": "centos/7",
     "ram": 512,
     "vcpu": 1,
     "ip_addr": "192.168.100.102",
            "rootPassWord": "abc123456789"
 },
 {
     "name": "hslave1",
     "box": "centos/7",
     "ram": 512,
     "vcpu": 1,
     "ip_addr": "192.168.100.103",
            "rootPassWord": "abc123456789"
 }
    ]
}

Prepare-cluster.py script:


Vagrantfile:

$ more Vagrantfile

# -*- mode: ruby -*-
# # vi: set ft=ruby :

# Specify minimum Vagrant version and Vagrant API version
Vagrant.require_version '>= 1.6.0'
VAGRANTFILE_API_VERSION = '2'

# Require JSON module
require 'json'
require 'pp'

# Read YAML file with box details
data = JSON.parse(File.read(File.join(File.dirname(__FILE__), 'servers.json')))

# Create boxes
Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

# config.ssh.username = 'root'
# config.ssh.password = 'vagrant'
# config.ssh.insert_key = 'true'


  servers = data["servers"]
  # Iterate through entries in JSON file
  servers.each do |server|
    pp server
    config.vm.define server['name'] do |srv|
      srv.vm.box = server['box']
      srv.vm.network 'private_network', ip: server['ip_addr']
      srv.vm.provider :vmware_fusion do |vmw|
        vmw.vmx['memsize'] = server['ram']
        vmw.vmx['numvcpus'] = server['vcpu']
      end # srv.vm.provider
    end # config.vm.define
  end # servers.each
end # Vagrant.configure

Execution steps:

1. gen-hosts-configs.py
2. set-blueprint-options.py
3. Optional: vagrant up (depends on the cluster type)
4. prepare-cluster.py
5. ansible
6. install_blueprints.py

Resources:
Ansible cluster configuration example:
https://github.com/gautamborad/hdp-ansible

Setting up /etc/hosts
http://www.rubydoc.info/gems/vagrant-hosts/2.6.0

Jinja 2 templates
http://docs.ansible.com/ansible/template_module.html

Use a Jinja 2 template with the hosts information
http://www.sebastien-han.fr/blog/2013/08/19/consistent-hosts-file-with-ansible/

#
127.0.0.1   localhost   {{ ansible_hostname }}
192.168.0.1     bla1.mydomain
192.168.0.2     bla2.mydomain
192.168.0.3     bla3.mydomain
192.168.0.4     bla4.mydomain
192.168.0.5     bla5.mydomain

The yaml file:

#
- hosts: all

  tasks:
      - name: template test
        action: template src=templates/hosts.j2 dest=/etc/hosts

OSX Ansible installation:
https://valdhaus.co/writings/ansible-mac-osx/

Or put Ansible in a virtualenv:
http://docs.python-guide.org/en/latest/dev/virtualenvs/
http://docs.ansible.com/ansible/intro_installation.html#latest-releases-on-mac-osx

Hortonworks prepare environment:
https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/prepare-environment.html

Installing the Ansible control machine:
http://docs.ansible.com/ansible/intro_installation.html#installing-the-control-machine

Ansible Ambari installation:
https://github.com/seanorama/ansible-ambari/blob/master/Vagrantfile

Ansible provisioning on Vagrant:
https://www.vagrantup.com/docs/provisioning/ansible.html

Script provisioning on Vagrant:
https://nicolasmaillard.com/2015/07/20/provisioning-virtual-dev-hdp-cluster-part-1-vagrant/

Automation blog:
http://henning.kropponline.de/2015/01/11/hdp-ansible-playbook-example/

Developing Ansible modules:
http://docs.ansible.com/ansible/developing_modules.html

Ansible Ambari module:
https://medium.com/@b23llc/managing-distributed-data-products-with-ansible-and-ambari-44c7175555d8#.5znvfj4bz

Ansible file layout:
http://leucos.github.io/ansible-files-layout/

nodes.rb:
https://github.com/hkropp/vagrant-hdp/blob/master/conf/nodes.rb
Example:

ansible playbook for ambari setup:
https://github.com/hkropp/vagrant-hdp/blob/master/ansible/hdp_centos6_playbook.yml
https://github.com/seanorama/ansible-ambari/blob/master/site.yml

Ambari in Ansible:
https://github.com/mbittmann/ambari-ansible-module

Ansible Directory Structure:
http://docs.ansible.com/ansible/playbooks_best_practices.html#directory-layout

Ansible Vault for secure passwords:
http://docs.ansible.com/ansible/playbooks_vault.html

Vagrant-cachier (box caching):
http://fgrehm.viewdocs.io/vagrant-cachier/

Blueprints:
https://github.com/hkropp/vagrant-hdp/blob/master/master_blueprint.json

https://medium.com/@b23llc/managing-distributed-data-products-with-ansible-and-ambari-44c7175555d8#.ytz5ceqsn

Some guest additions were not installed (do not ignore this message):
Fix: https://github.com/geerlingguy/drupal-vm/issues/175

#
Error:
==> master1: Checking for guest additions in VM...
    master1: No guest additions were detected on the base box for this VM! Guest
    master1: additions are required for forwarded ports, shared folders, host only
    master1: networking, and more. If SSH fails on this machine, please install
    master1: the guest additions and repackage the box to continue.
    master1: 
    master1: This is not an error message; everything may continue to work properly,

    master1: in which case you may ignore this message.

Ansible hints:
http://leucos.github.io/ansible-files-layout/