Wednesday, March 30, 2016

Atom text editor: Configuration for OSX and setup for Python development

Setup
Mac download the dmg and install it in Applications

Create a Python virtualenv to install Atom depdencies in and to launch Atom from
Reference: https://virtualenv.pypa.io/en/latest/
From: http://www.marinamele.com/2014/05/install-python-virtualenv-virtualenvwrapper-mavericks.html

Start the virtualenv and cd to the virtual env with the Atom dependencies installed (you will install them below if you want).

Now launch Atom from inside of the virtualenv
$ /Applications/Atom.app/Contents/MacOS/Atom

Make an alias for launching Atom inside of the activated virtualenv

alias atom_mac="workon atom_editor_env; cd ~/.virtualenvs/atom_editor_env; /Applications/Atom.app/Contents/MacOS/Atom &"

Ubuntu-one way to do it
http://itsfoss.com/install-atom-text-editor-ubuntu-1404-linux-mint-17/

Atom Editor Keyboard Shortcut Cheat Sheets

Mac OSX key notes
Alt in the Atom docs is option on the Mac keyboard

Atom Python configuration
Install packages in Atom using the directions from the link below
From: http://www.marinamele.com/install-and-configure-atom-editor-for-python

Atom package installation list
Note: install all Python packages in a virtualenv (see above for the setup instructions)
https://atom.io/packages/box-edit
Alt-S to enable
Use the mouse to select
Cntl-{c,v}
Cntl-x does not seem to work in some cases

https://atom.io/packages/linter-flake8
pip install flake8
pip install flake8-docstrings
pip install hacking
You will need to open the Atom init script (Atom –> Open your Init Script) 

http://flight-manual.atom.io/
http://flight-manual.atom.io/hacking-atom/sections/the-init-file/

Add the following to the ~/.atom/init.coffee file:
Crayon Syntax Highlighter v2.7.1 # config option currently does not load correctly- do not use
process.env.PATH = ['/usr/local/bin/', process.env.PATH].join(':')


Project configuration files
Linter also supports reading flake8 configuration files. To use them, you need specify their names manually into Config Files Names in the package settings. Note that package settings from Settings panel will be ignored if a configuration file was found.

https://atom.io/packages/linter-pep8
pip install flake8
pip install flake8-docstrings

https://atom.io/packages/autocomplete-python

atom.io/packages/auto-indent
From: http://stackoverflow.com/questions/22611377/auto-indent-code-in-atom-editor
Package auto-indent exists to apply auto-indent to entire file with this shortcuts :
ctrl+shift+i
or
cmd+shift+i
Package url : atom.io/packages/auto-indent

https://atom.io/packages/linter
https://atom.io/packages//python-debugger
  1. Press alt-r (OSX: Option-r) to show the debugger view
  2. Insert input arguments in the input arguments field if applicable
  3. Hit the Run button. Focus moves to the first breakpoint.
https://atom.io/packages/language-python

https://atom.io/packages/python-tools

https://atom.io/packages/minimap
Minimap: Displays a small map of the current file on the right side of your document (like Sublime
Text by default).
Atom Tutorials
http://code.tutsplus.com/tutorials/check-out-atom-githubs-new-development-editor--net-37030
See the code snippets section

Using Atom as a Python Editor
http://pythonhow.com/using-atom-as-a-python-editor/


UNDER CONSTRUCTION!!
Lets build a script to install Atom with the python configuration on new machines.
1. Install apm

2.Install each of the atom packages above with apm
apm install atom-beautify
...
3. Create the python virutalenv

4. Add the alias to .bashrc

Now lets use Atom to edit Python files:
http://pythonhow.com/using-atom-as-a-python-editor/


Tuesday, March 29, 2016

Installing maven2 without openjdk

How to install Oracle JDK on Ubuntu 16.04

https://community.linuxmint.com/tutorial/view/1372

http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

# Download the 64-bit version if you have a 64-bit machine
Linux x64
#174.76 MB  jdk-8u121-linux-x64.tar.gz

# Check the md5 number with md5sum and compare it to the sum on the Oracle site

sha256sum  jdk-8u121-linux-i586.tar.gz

tar xvf jdk-8u121-linux-i586.tar.gz

# Lets improve the installation process by setting the java version in a variable so the user can change the version in one place.

java_version=1.8.0_121
sudo mkdir jdk
sudo mv jdk$java_version /opt/java

# 8. Make JDK system default.

sudo update-alternatives --install "/usr/bin/java" "java" "/opt/java/jdk$java_version/bin/java" 1
sudo update-alternatives --set java /opt/java/jdk$java_version/bin/java

# 9. Test installed java version.

$ java -version
java version "$java_version" Java(TM) SE Runtime Environment (build $java_version-b15) Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)

# Optinaly install firefox plugin by linking libnpjp2.so file to ~/.mozzila/plugins folder.

mkdir -p ~/.mozzila/plugins
ln -s /opt/java/jdk$java_version/jre/lib/amd64/libnpjp2.so ~/.mozzila/plugins/

Make a symbolic link to point to the specific java version and reference the symbolic link in your .bashrc files. If you save your bashrc files in Dropbox or some other cloud provider and then link to the cloud version of the .bashrc files then the JAVA_HOME will always point to the symlink. This is important because if you use different computers they each might have a different version of Java 1.7 for example but JAVA_HOME will point to the symlink which will point to the Java version on the specific computer.

$ sudo mkdir /usr/lib/jvm
$ sudo ln -s  /opt/java/jdk1.7.0_79/ /usr/lib/jvm/java-7-oracle
 

~/.bashrc:export JAVA_HOME="/usr/lib/jvm/java-7-oracle"




Switching easily between Java JDKs on Mac OS X

From: https://wimdeblauwe.wordpress.com/2014/03/20/switching-easily-between-java-jdks-on-mac-os-x/

First, add the following aliases in ~/.profile file:
alias setJdk6='export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)'
alias setJdk7='export JAVA_HOME=$(/usr/libexec/java_home -v 1.7)'
alias setJdk8='export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)'
Open a new terminal window or use source ~/.profile to load the aliases.
Now you can easily switch between the installed JDKs:
> setJdk6
We can verify that Maven picks this up correctly:
> mvn --version
Apache Maven 3.0.4 (r1232337; 2012-01-17 09:44:56+0100)
Maven home: /Applications/maven/apache-maven-3.0.4
Java version: 1.6.0_65, vendor: Apple Inc.
Java home: /System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home
Default locale: en_US, platform encoding: MacRoman
OS name: "mac os x", version: "10.9.2", arch: "x86_64", family: "mac"

Sunday, March 27, 2016

mvn dependency management notes

http://stackoverflow.com/questions/5805962/how-can-i-locate-the-pom-which-requests-a-missing-pom

mvn dependency:analyze-dep-mgt


mvn dependency:tree -Dverbose=true can spit out unused duplicates/versions. Wasn't enough in my case, however, for some reason it didn't show the offending jar's listed [?]. It also doesn't show the ommitted jar's descendants, which may or may not be useful.

Saturday, March 26, 2016

Jenkins and Nexus on OSX

https://wiki.jenkins-ci.org/display/JENKINS/Thanks+for+using+OSX+Installer
Read the part about Apple Java...

https://wiki.jenkins-ci.org/display/JENKINS/Thanks+for+using+OSX+Installer

Add some handy aliases to control Jenkins to your .bash_profile
alias jenkins_off='sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist'

alias jenkins_on='sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist'

Jenkins uses a user named "jenkins" and on the Mac the jenkins home dir is /Users/Shared/Jenkins/Home.

Log in as jenkins from the command line
$ sudo su jenkins -
(no password required)

Setup the jenkins password
sudo passwd jenkins

There are other ways to set up Jenkins and git access but this is the easiest way I have found to do so. 

Create new ssh key that you will use to access git to /Users/Shared/Jenkins/.ssh
ssh-keygen
(DO NOT ADD A PASSWORD TO THE KEY)

Copy the public key to github in your profile


In the following Jenkins page create a new credential with your git username, the path to the ssh file in Jenkins, and NO password



Now make a new project or go to an existing project
Jenkins > > configuration

Source Code Management
Check if git can access the repo

git -c core.askpass=true ls-remote -h HEAD
(returns nothing on success)

**
http://obscuredclarity.blogspot.com/2012/04/continuous-integration-using-jenkins.html
Set up Jenkins security using the instructions from above.

Select this option


Skip to the instructions for the Java and Maven installation below at ****.

Set up the rest of Jenkins using the instructions above @**

****
https://gepreemacdev.wordpress.com/2012/07/20/step-configure-jenkins/
Note: set the MAVEN_HOME to:  /usr/local/Cellar/maven/3.2.5
(Not  /usr/local/Cellar/maven/3.2.5/libexec)
Change the version to match your systems maven version.


http://stackoverflow.com/questions/13512948/configuring-jenkins-email-notification-with-gmail
configure Jenkins to send email

From: https://support.google.com/accounts/answer/6010255
Below are the updated gmail SMTP server configuration :
Gmail SMTP server address: smtp.gmail.com
Gmail SMTP user name: Your full Gmail address (e.g. example@gmail.com)
Gmail SMTP password: Your Gmail password
Gmail SMTP port: 465
Gmail SMTP TLS/SSL required: yes
This configuration is working fine for me .
The above configuration didn't work becase of added security measures by google. An additional configuration is required in the Gmail account from which the mail is being triggered. This step authorizes apps like Jenkins to use the Gmail account.

https://support.google.com/accounts/answer/6010255

After this emails were triggered successfully

http://books.sonatype.com/nexus-book/reference/

Friday, March 25, 2016

More notes: Similarity measurement

https://janav.wordpress.com/2013/10/27/tf-idf-and-cosine-similarity/

From:
http://stackoverflow.com/questions/24011418/apache-spark-naive-bayes-based-text-classification

For example i have two types of Class (C and J)
Train data is :
C, Chinese Beijing Chinese
C, Chinese Chinese Shanghai
C, Chinese Macao
J, Tokyo Japan Chinese
And test data is : Chinese Chinese Chinese Tokyo Japan // What is ist J or C ?
...
But you can do it manually by first creating a dictionary of terms. Then compute IDFs for each term and then convert each documents into vectors using the TF-IDF scores.
...
There any many classification methods (logistic regression, SVMs, neural networks,LDA, QDA...), you can either implement yours or use MLlib classification methods (actually, there are logistic regression and SVM implemented in MLlib)
What you need to do is transform your features to a vector, and labels to doubles.
For examples, your dataset will look like:
1, (2,1,0,0,0,0)
1, (2,0,1,0,0,0)
1, (1,0,0,1,0,0)
0, (1,0,0,0,1,1)
And tour test vector:
(3,0,0,0,1,1)
From:



From:
Another method for creating recs with Spark is the search engine method. This is basically a cooccurrence recommender served by Solr or Elasticsearch. Comparing factorized to cooccurrence is beyond this question so I'll just describe the latter.
You feed interactions (user-id,item-id) into Mahout's spark-itemsimilarity. This produces a list of similar items for every item seen in the interaction data. It will come out by default as a csv and so can be stored anywhere. But it needs to be indexed by a search engine.
...
One other benefit of either factorization or the search method is that entirely new users and new history can be used to create recs where the older Mahout recommenders could only recommend to users and interactions known when the job was run.
Descriptions here:






Machine Learning Notes...

Installing Java 8 on OSX

Download instructions
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

https://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jdk.html#A1096855

Find the Java Control Panel on Mac

Launch the Java Control Panel on Mac OS X (10.7.3 and above)
  1. Click on Apple icon on upper left of screen.
  2. Go to System Preferences
  3. Click on the Java icon to access the Java Control Panel.

Improved how to switch Java versions using BASH:


  1. Find the Java version that is required:
    1.  /usr/libexec/java_home  -V
  2. Run the java.sh script below:
    1. java.sh
    2. Example:
$ /usr/libexec/java_home  -V
Matching Java Virtual Machines (3):
    1.8.0_77, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home
    1.8.0_60, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home
    1.7.0_71, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home

/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home

$ setjdk 1.8.0_77


https://github.com/StevenACoffman/dotfiles/blob/master/bin/shell/functions/java.sh

The Java version switcher scripts use java_home program.
http://docs.oracle.com/javase/8/docs/technotes/guides/install/mac_jdk.html#A1096903

$ /usr/libexec/java_home  --help
Usage: java_home [options...]
    Returns the path to a Java home directory from the current user's settings.

Options:
    [-v/--version   ]       Filter Java versions in the "JVMVersion" form 1.X(+ or *).
    [-a/--arch      ]  Filter JVMs matching architecture (i386, x86_64, etc).
    [-d/--datamodel ]     Filter JVMs capable of -d32 or -d64
    [-t/--task      ]          Use the JVM list for a specific task (Applets, WebStart, BundledApp, JNI, or CommandLine)
    [-F/--failfast]                  Fail when filters return no JVMs, do not continue with default.
    [   --exec      ...]   Execute the $JAVA_HOME/bin/ with the remaining arguments.
    [-R/--request]                   Request installation of a Java Runtime if not installed.
    [-X/--xml]                       Print full JVM list and additional data as XML plist.
    [-V/--verbose]                   Print full JVM list with architectures.
    [-h/--help]                      This usage information.

$ /usr/libexec/java_home  -
/Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home

$ /usr/libexec/java_home  -V
Matching Java Virtual Machines (3):
    1.8.0_77, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_77.jdk/Contents/Home
    1.8.0_60, x86_64: "Java SE 8" /Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home
    1.7.0_71, x86_64: "Java SE 7" /Library/Java/JavaVirtualMachines/jdk1.7.0_71.jdk/Contents/Home

Thursday, March 24, 2016

Adding JPS to Ubuntu

 yum install java-1.6.0-openjdk-devel.x86_64

Wednesday, March 23, 2016

Automating copying SSH keys to multiple servers in a cluster

Lets make a system for configuring the /etc/hosts and ssh keys on the servers

1. Make the ssh key
http://stackoverflow.com/questions/2848725/how-can-i-ssh-inside-a-perl-script


2. Read a list of pairs of ip addresses and host names from a CSV file.

hosts.csv
192.168.0.100,centos0
192.168.0.101,centos1
192.168.0.102,centos2
192.168.0.103,centos3
192.168.0.104,centos4

Under construction

Add current working directory to @INC to load the modules that we will create below

http://www.perlmonks.org/?node_id=375341  
use lib '.';
 
From: http://www.tutorialspoint.com/perl/perl_oo_perl.htm

#!/usr/bin/perl

package ServerInfo;

sub new
{
    my $class = shift;
    my $self = {
        _hostName  => shift,
        _ipAddress => shift,
        _password  => shift,

    };
# Print all the values just for clarification.
    print "hostName is $self->{_hostName}\n";
    print "ipAddress is $self->{_ipAddress}\n";
    bless $self, $class;
    return $self;
}

sub getHostName {
    my( $self ) = @_;
    return $self->{_hostName};
}

sub getIpAddress {
    my( $self ) = @_;
    return $self->{_ipAddress};
}

sub getPassword {
    my( $self ) = @_;
    return $self->{_password};
}

1;




Now create Person object in the config.pl file as follows:

#!/usr/bin/perl

use ServerInfo;

if ($#ARGV != 0) {
   print("Args: ");
   exit;
}

$passWord = $ARGV[0];

$object = new ServerInfo( "192.168.0.100", "centos0",$passWord);

$hostName = $object-> getHostName();
print "HostName is : $hostName\n";

$ipAddress = $object-> getIpAddress();
print "ipAddress is : $ipAddress\n";


Import the host file.

#!/usr/bin/perl

#!/usr/bin/perl use strict; use warnings;   use Text::CSV; my $csv = Text::CSV->new({ sep_char => ',' });   my $file = $ARGV[0] or die "Need to get CSV file on the command line\n";   my $sum = 0; open(my $data, '<', $file) or die "Could not open '$file' $!\n"; while (my $line = <$data>) { chomp $line;   if ($csv->parse($line)) {   my @fields = $csv->fields(); $sum += $fields[2];   } else { warn "Line could not be parsed: $line\n"; } } print "$sum\n";



3. Add the new hosts information to the /etc/hosts file




4. Add the generated ssh key to the remote servers



cat ~/.ssh/id_rsa.pub | ssh root@HOST "mkdir -p ~/.ssh && cat >> ~/.ssh/authorized
 
chmod 700 ~/.ssh

chmod 600 ~/.ssh/authorized_keys 
 
#Restart sshd on the remote host
 
sudo /etc/init.d/sshd restart < $password
 
Now test the passwordless ssh login to root@host 


5. Test the ssh passwordless login to the remote servers










Saturday, March 19, 2016

Different VM types and Docker and Vagrant

http://stackoverflow.com/questions/16647069/should-i-use-vagrant-or-docker-for-creating-an-isolated-environment

"I'm the author of Docker.
The short answer is that if you want to manage machines, you should use Vagrant. And if you want to build and run applications environments, you should use Docker."
https://www.quora.com/What-is-the-difference-between-Docker-and-Vagrant-When-should-you-use-each-one

"The limitations of containers vs VM should be obvious now: You can't run completely different OS in containers like in VMs. However you can run different distros of Linux because they do share the same kernel. The isolation level is not as strong as in VM. In fact, there was a way for "guest" container to take over host in early implementations. Also you can see that when you load new container, the entire new copy of OS doesn't start like it does in VM. All containers share same kernel. This is why containers are light weight. Also unlike VM, you don't have to pre-allocate significant chunk of memory to VM because we are not running new copy of OS. This enables to run thousands of containers on one OS while sandboxing them which might not be possible to do if we were running separate copy of OS in its own VM."
"In summary: Vagrant is for managing machines, Docker is for building and running application environments."

"Vagrant was designed to manage virtual machines. Docker was designed to manage an application runtime. This means that Docker, by design, can interact with an application in richer ways, and has more information about the application runtime. The primitives in Docker are processes, log streams, environment variables and network links between components. The primitives in Vagrant are machines, block devices, and ssh keys. Vagrant simply sits lower in the stack, and the only way it can interact with a container is by pretending it's just another kind of machine, that you can "boot" and "log into"."
Container vs VM

https://www.docker.com/what-docker

"Lightweight

Containers running on a single machine all share the same operating system kernel so they start instantly and make more efficient use of RAM. Images are constructed from layered filesystems so they can share common files, making disk usage and image downloads much more efficient."
https://www.docker.com/use-cases/continuous-integration

http://superuser.com/questions/889472/docker-containers-have-their-own-kernel-or-not

http://web.archive.org/web/20150326185901/http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part

https://github.com/opencontainers/runc



http://www.projectatomic.io/docs/filesystems/

"Supported Filesystems

A core part of the Docker model is the efficient use of layered images and containers based on images. To implement this Docker relies heavily on various filesystem features in the kernel. This document will explain how this works and give some advice in how to best use it"
http://superuser.com/questions/889472/docker-containers-have-their-own-kernel-or-not

https://www.upguard.com/articles/docker-vs-vagrant

http://stackoverflow.com/questions/16047306/how-is-docker-different-from-a-normal-virtual-machine?rq=1
"A full virtualized system usually takes minutes to start, LXC containers take seconds, and sometimes even less than a second.
There are pros and cons for each type of virtualized system. If you want full isolation with guaranteed resources, a full VM is the way to go. If you just want to isolate processes from each other and want to run a ton of them on a reasonably sized host, then LXC might be the way to go."

http://stackoverflow.com/questions/16647069/should-i-use-vagrant-or-docker-for-creating-an-isolated-environment

https://www.linux.com/news/software/applications/735734-forum-thread-a-comparison-of-vagrant-and-docker-

https://en.wikipedia.org/wiki/LXC

https://en.wikipedia.org/wiki/Virtual_machine

https://linuxcontainers.org/

https://linuxcontainers.org/lxc/news/

https://github.com/opencontainers/runc

http://web.archive.org/web/20150326185901/http://blog.dotcloud.com/under-the-hood-linux-kernels-on-dotcloud-part