CENTOS: YUM multi machine caching links

http://yum.baseurl.org/wiki/YumMultipleMachineCaching

http://pulpproject.org/demo/

CENTOS: yum being replaced by dnf

https://www.vultr.com/docs/use-dnf-to-manage-software-packages-on-centos-7

http://dnf.baseurl.org/

PC maintenance

1. Check the CPU temperature in the BIOS

2. Change the CPU thermal paste
-check the CPU temperature again. Should be lower. If not then consider installing a new CPU fan.

3. Use an air compressor to blow the dust out of the case and the power supply

4. Run diagnostics to determine if the CPU, motherboard, memory and power supply are working correctly.

5. Upgrade the BIOS
-start the computer
-enter the BIOS
-check the MB and BIOS version at the top of the screen
-go to the MB vendor's web page
-find the latest version of the MB's BIOS and download if it is more recent than the MB BIOS
-Upgrade the BIOS
--start the computer
--enter the BIOS
--go to the maintenance tab
--flash the BIOS using the

Neural nets: word2vec

http://deeplearning4j.org/word2vec#intro

https://en.wikipedia.org/wiki/Word2vec

https://www.quora.com/How-does-word2vec-work

http://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf

http://arxiv.org/pdf/1402.3722v1.pdf

https://levyomer.files.wordpress.com/2014/09/neural-word-embeddings-as-implicit-matrix-factorization.pdf

Neural nets: preprocessing text

http://www.coling-2014.org/COLING%202014%20Tutorial-fix%20-%20Tomas%20Mikolov.pdf

http://mt-class.org/jhu/slides/lecture-nn-lm.pdf

http://crl.ucsd.edu/~elman/Papers/fsit.pdf

https://www.kaggle.com/c/word2vec-nlp-tutorial/details/part-2-word-vectors

https://cs.stanford.edu/~quocle/paragraph_vector.pdf

https://visualstudiomagazine.com/articles/2013/07/01/neural-network-data-normalization-and-encoding.aspx

https://en.wikipedia.org/wiki/Topic_model

https://en.wikipedia.org/wiki/Language_model

http://datascience.stackexchange.com/questions/869/neural-network-parse-string-data

https://code.google.com/archive/p/word2vec/

http://web.stanford.edu/group/pdplab/pdphandbook/handbookch8.html

Related topics

http://www.cs.toronto.edu/~ilya/pubs/2011/LANG-RNN.pdf

https://deeplearningworkshopnips2011.files.wordpress.com/2011/12/21.pdf

Neural nets: Link to CUDNN installation instructions for Linux

http://www.pyimagesearch.com/2016/07/04/how-to-install-cuda-toolkit-and-cudnn-for-deep-learning/

More neural network references

https://www.analyticsvidhya.com/blog/2016/03/introduction-deep-learning-fundamentals-neural-networks/

Scala: Trees: binary, sorted, tail recursive

Scala: Tail recursive tree traversal
http://www.scala-lang.org/old/node/7984

Scala Tree Sort Examplehttps://www.garysieling.com/blog/scala-tree-sort-example
https://dzone.com/articles/scala-tree-sort-example

Binary tree
http://www.scala-lang.org/old/node/52.html

Scala IDE Tutorial link

https://app.assembla.com/wiki/show/scala-ide/Tutorial

Scala ArrayBuffer documentation

http://www.scala-lang.org/api/2.8.1/scala/collection/mutable/ArrayBuffer.html

Scala traits, objects and classes

http://www.artima.com/pins1ed/traits.html#12

http://www.tutorialspoint.com/scala/scala_classes_objects.htm

Scala: Instrumented Quicksort

It is some times a lot easier to see what an algorithm is doing if print statements are added to the code. Note that each time that the sort function is called recursively the level counter is incremented. the printout is indented one more space for each level (recursion call).

$ scalac -feature MySorter.scala 

$ scala -classpath . MySorter

my sorter

 [1] in: ( ) 6 3 5 2 7 4 8 1

 [1] pivot:  ( ) 7

  [2] in: (>) 6 3 5 2 4 1

  [2] pivot:  (>) 2

   [3] in: (>) 1

   [3] out: (>) 1

   [3] in: (<) 6 3 5 4

   [3] pivot:  (<) 5

    [4] in: (>) 3 4

    [4] pivot:  (>) 4

     [5] in: (>) 3

     [5] out: (>) 3

     [5] in: (<) 

     [5] out: (<) 

   [3] out: (>) 3 4

    [4] in: (<) 6

    [4] out: (<) 6

  [2] out: (<) 3 4 5 6

 [1] out: (>) 1 2 3 4 5 6

  [2] in: (<) 8

  [2] out: (<) 8

[0] out: ( ) 1 2 3 4 5 6 7 8

[0] final: s1 2 3 4 5 6 7 8

// Original code from http://www.scala-lang.org/docu/files/ScalaByExample.pdf

def sort(xs: Array[Int]): Array[Int] = {
  if (xs.length <= 1) xs
  else {
     val pivot = xs(xs.length / 2)
     Array.concat( sort(xs filter (pivot >)), xs filter (pivot ==), sort(xs filter (pivot <)))
  }
}

// Modified code with instrumentation
// compile the code below

import scala.language.postfixOps

object MySorter {

  def indent(len : Int) : String = {
    var sb = new String

    for (i <- 1="" len="" p="" to="">      sb += " "
    }

    return sb
  }

  def printArray[T](level : Int, msg : String, a : Array[T]) {
    val s = a.mkString(" ")
    val padding = indent(level)
    println(padding + "[" + level +  "] "  + msg + s)
  }

  def sort(xs: Array[Int], level : Int, comparison : String): Array[Int] = {
    var l = level
    l += 1

    val padding = indent(l)

    printArray(l, "in: (" + comparison + ") ", xs)
    if (xs.length <= 1) {
      printArray(l, "out: (" + comparison + ") ", xs)
      xs
    }
    else {
      val pivot = xs(xs.length / 2)
      println(padding + "[" + l +  "]"  + " pivot:  (" + comparison + ") " + pivot)
      var out = Array.concat(
        sort(xs filter (pivot >), l, ">"),
        xs filter (pivot ==),
        sort(xs filter (pivot <),l, "<"))
      l -= 1
      printArray(l, "out: (" + comparison + ") ", out)
      return out
    }
  }

  def main(args: Array[String]) {
    println("my sorter")
    var a = Array(6, 3, 5, 2, 7, 4, 8,1)
    var level = 0
    var aa = sort(a, level, " ")
    //printArray(a)
    printArray(0, "final: s", aa)
  }
}

Scala infix operators: nice Stackoverflow explaination

http://stackoverflow.com/questions/24681777/scala-value-is-not-a-member-of-int

Scala enumerations

http://underscore.io/blog/posts/2014/09/03/enumerations.html

https://gist.github.com/chaotic3quilibrium/57add1cd762eb90f6b24

http://stackoverflow.com/questions/1898932/case-objects-vs-enumerations-in-scala

http://stackoverflow.com/questions/17646335/combining-enums-and-using-getter-to-return-a-specified-enum

http://jackcoughonsoftware.blogspot.rs/2008/06/scala-and-enums.html

http://stackoverflow.com/questions/26492941/scala-enumeration-with-constructor-and-lookup-table

http://stackoverflow.com/questions/4346580/how-to-add-a-method-to-enumeration-in-scala

object Suit extends Enumeration {
   val Clubs, Diamonds, Hearts, Spades = Value

   class SuitValue(suit: Value) {
      def isRed = !isBlack
      def isBlack = suit match {
         case Clubs | Spades => true
         case _              => false
      }
   }

   implicit def value2SuitValue(suit: Value) = new SuitValue(suit)
} 

object Direction extends Enumeration { 

  type Direction = Value

  val NORTH, SOUTH, EAST, WEST = Value

}

import Direction._




var value: Set[Direction.Value] = Set[Direction.Value](Direction.NORTH, Direction.SOUTH)

Scala enumerations

http://underscore.io/blog/posts/2014/09/03/enumerations.html

https://gist.github.com/chaotic3quilibrium/57add1cd762eb90f6b24

http://stackoverflow.com/questions/1898932/case-objects-vs-enumerations-in-scala

http://stackoverflow.com/questions/17646335/combining-enums-and-using-getter-to-return-a-specified-enum

Scala:Monads

http://james-iry.blogspot.rs/2007/09/monads-are-elephants-part-1.html

http://james-iry.blogspot.rs/2007/10/monads-are-elephants-part-2.html

http://james-iry.blogspot.rs/2007/10/monads-are-elephants-part-3.html

http://james-iry.blogspot.rs/2007/11/monads-are-elephants-part-4.html

http://scabl.blogspot.rs/2013/02/monads-in-scala-1.html

Scala: Graphs

http://blog.higher-order.com/

https://github.com/oncue/quiver

https://oncue.github.io/quiver/

http://web.engr.oregonstate.edu/~erwig/fgl/haskell/

https://bintray.com/oncue/releases/quiver/

More Scala URL's

https://oldfashionedsoftware.com/category/scala/

http://www.tutorialspoint.com/scala/scala_traits.htm

http://joelabrahamsson.com/programming/scala/

Scalaz

http://timperrett.com/2014/07/20/scalaz-task-the-missing-documentation/

https://www.infoq.com/presentations/Scalaz-Functional-Programming-in-Scala

https://hseeberger.wordpress.com/2010/11/25/introduction-to-category-theory-in-scala/

https://hseeberger.wordpress.com/2011/01/31/applicatives-are-generalized-functors/

http://eed3si9n.com/learning-scalaz/

https://www.infoq.com/presentations/Scalaz-Functional-Programming-in-Scala

A post about nothing in Scala

The post at the following URL explains nothingness in Scala.

https://oldfashionedsoftware.com/2008/08/20/a-post-about-nothing/

From the URL:

"The method getAStringMaybe returns Option[String]. Option is an abstract class with exactly two subclasses, class Some and object None. Those are the only two ways to instantiate an Option. So getAStringMaybe returns either a Some[String] or None. Some and None are case classes, so you can use the handy match/case construct to handle the result. None is object that signifies no result from the method.

The purpose of an Option[T] return type is to tell callers that the method might return a T in the form of a Some[T], or it might return None to signify no result. This way, the caller supposedly knows when he does and does not need to check for a good return value."

Unit is the same as void in Java

"This is another easy one. Unit is the type of a method that doesn’t return a value of any sort. Sound familiar? It’s like a void return type in Java. Here’s an example:

scala> def doThreeTimes(fn: (Int) => Unit) = {
| fn(1); fn(2); fn(3);
| }
doThreeTimes: ((Int) => Unit)Unit"

"One other use of Nothing is as a return type for methods that never return. It makes sense if you think about it. If a method’s return type is Nothing, and there exists absolutely no instance of Nothing, then such a method must never return.

Your logging method would return Unit. There is a value Unit so it can actually be returned. From the API docs:

Unit is a subtype of scala.AnyVal. There is only one value of type Unit, (), and it is not represented by any object in the underlying runtime system. A method with return type Unit is analogous to a Java method which is declared void."

http://stackoverflow.com/questions/26563213/how-to-instantiate-unit-in-scala

You can just use () whose type is Unit:

scala> import scala.collection.mutable.HashMap
import scala.collection.mutable.HashMap

scala> val myMap = new HashMap[String, Unit]()
myMap: scala.collection.mutable.HashMap[String,Unit] = Map()

scala> myMap + ("myStringKey" -> ())
res1: scala.collection.mutable.Map[String,Unit] = Map(myStringKey -> ())

This is a comment taken from Unit.scala:

There is only one value of type Unit, (), and it is not represented by any object in the underlying runtime system.

Mac OSX Hadoop setup

Setup JAVA_HOME

http://stackoverflow.com/questions/22842743/setting-java-home-environment-variable-on-mac-osx-10-9

echo export "JAVA_HOME=\$(/usr/libexec/java_home)" >> ~/.bash_profile

and restart your shell.

If you have multiple JDK versions installed and you want it to be a specific one, you can use the -v flag to java_home like so:

echo export "JAVA_HOME=\$(/usr/libexec/java_home -v 1.7)" >> ~/.bash_profile

https://amodernstory.com/2014/09/23/installing-hadoop-on-mac-osx-yosemite/

Change the version from 2.6.0 to 2.7.2 or what ever the latest version is

Make a user account on HDFS (use your user name...)
https://docs.hortonworks.com/HDPDocuments/Ambari-2.1.2.0/bk_ambari_views_guide/content/_setup_HDFS_user_directory.html


Now run an example Map Reduce job (change the commands to use the /user/ that you created above):

https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Example:_WordCount_v1.0

http://stackoverflow.com/questions/22726305/get-error-when-i-run-hbase-shell

. Following the instructions here, before run ./bin/hbase shell command you should use ./bin/start-hbase.sh first. Then my problem was solved.

Scala implicits

http://debasishg.blogspot.com.es/2010/06/scala-implicits-type-classes-here-i.html

Scala Unit Explaination

http://stackoverflow.com/questions/14205367/found-unit-required-int-why-is-the-error-not-obvious

From the above link

37down voteaccepted

I'll first explain what type Unit is, just in case. Even if you know already, other people having the same kind of problem might well not know it. Type Unit is similar to what is known in C or Java as void. In those languages, that means "this does not return anything". However, every method in Scala returns a value. To bridge the gap between every method returning something and not having anything useful to return, there's Unit. This type is an AnyVal, which means it isn't allocated on the heap, unless it gets boxed or is the type of a field on an object. Furthermore, it has only one value, whose literal is (). That is, you could write this: val x: Unit = () The practical effect of this is that, when a method "returns" Unit, the compiler doesn't have to actually return any value, since it already knows what that value is. Therefore, it can implement methods returning Unit by declaring them, on the bytecode level, as being void. Anyway, if you don't want to return anything, you return Unit. Now let's look at the code given. Eclipse says it returns Unit, and, in fact, Eclipse is correct. However, most people would actually make the error of having that method return AnyVal or Any, not Unit. See the following snippet for an example: scala> if (true) 2 res0: AnyVal = 2 So, what happened? Well, when Scala finds an if statement, it has to figure out what is the type returned by it (in Scala, if statements return values as well). Consider the following hypothetical line: if (flag) x else y Clearly, the returned value will be either x or y, so the type must be such that both x and y will fit. One such type is Any, since everything has type Any. If both x and y were of the same type -- say, Int -- then that would also be a valid return type. Since Scala picks the most specific type, it would pick Int over Any. Now, what happens when you don't have an else statement? Even in the absence of an elsestatement, the condition may be false -- otherwise, there would be no point in using if. What Scala does, in this case, is to add an else statement. That is, it rewrites that if statement like this: if (true) 2 else () Like I said before: if you do not have anything to return, return Unit! That is exactly what happens. Since both Int and Unit are AnyVal, and given that AnyVal is more specific than Any, that line returns AnyVal. So far I have explained what others might have seen, but not what happens in that specific code in the question: if (Tokens.KEYWORDS.contains(sfType)) { val retVal = TokenTypes.RESERVED_WORD } We have seen already that Scala will rewrite it like this: if (Tokens.KEYWORDS.contains(sfType)) { val retVal = TokenTypes.RESERVED_WORD } else () We have also seen that Scala will pick the most specific type between the two possible results. Finally, Eclipse tells use that the return type is Unit, so the only possible explanation is that the type of this: val retVal = TokenTypes.RESERVED_WORD is also Unit. And that's precisely correct: statements that declare things in Scala have type Unit. And, so, by the way, do assignments. The solution, as others have pointed out, is to remove the assignment and add an else statement also returning an Int: def contains1(sfType: TokenType): Int = if (Tokens.KEYWORDS.contains(sfType)) TokenTypes.RESERVED_WORD else -1

Scala style guide

https://github.com/databricks/scala-style-guide

Databricks Scala Style Guide

https://github.com/databricks/scala-style-guide

Yet another revision control system

Git is new to me. Git is probably the 5th or 6th revision control system that I have used (i.e. sccs, rcs, ...).

Setting up git merge and git diff using opendiff
http://stackoverflow.com/questions/21486481/is-it-possible-to-view-git-diffs-using-a-gui-side-by-side-tool-on-mac

how to install a home grown Google Chrome extension on OSX or Windows

Make temporary directory

mv the crx file to the temporary directory

Unzip the crx file

http://www.howtogeek.com/233355/how-to-install-extensions-from-outside-the-chrome-web-store-and-firefox-add-ons-gallery/

To do this, open the Extensions page — click the menu button, point to “More tools”, and select “Extensions”. Click the “Developer mode” checkbox to activate it, and then click the “Load unpacked extension” button. Navigate to the extension’s directory and open it.

http://www.howtogeek.com/191364/how-do-you-re-enable-non-web-store-extensions-in-the-stable-and-beta-channels-of-chrome/

Developer Mode Route

1. Download the crx file and unpack the extension using your favorite decompresser. Take note of the directory where you placed it.

2. Open the extension page and activate “Developer Mode”.

3. Click “Load unpacked extension…”

4. Search through your directory tree for the location where you unpacked your extension and click OK. If your extension is called “my extension”, then select the “my extension” directory.

Advantages: You do not have to install anything else.

Disadvantages: Chrome nags you to disable the extension every time you start it up.

How to run a Python script in a virtualenv

http://stackoverflow.com/questions/11963019/running-python-script-from-inside-virtualenv-bin-is-not-working

Or another option is to simply create a tiny bash wrapper that calls your original pythons script, which will allow you to leave your original script with a generic she-bang..

So if myscript.py is: #!/usr/bin/env python ...

Then you can make a myscript :

#!/bin/bash

/Users/foo/environments/project/env/bin/python myscript.py

When you do myscript, it will explicitly call your python script with the interpreter you set up.

If you have to pass args to the lower level script:

#!/bin/bash

echo $@

/Users/foo/environments/project/env/bin/python myscript.py $@