programming matrix: April 2016

Tuesday, April 26, 2016

Vagrant DNS Configuration

https://github.com/BerlinVagrant/vagrant-dns/

Wednesday, April 20, 2016

SPARK URL's

http://spark.apache.org/docs/latest/programming-guide.html

http://www.openkb.info/2015/01/scala-on-spark-cheatsheet.html

https://www.linkedin.com/pulse/spark-mapreduce-scala-underscore-vishnu-viswanath

val lines = sc.parallelize(List("a b", "c d","e f","g h","i j","k l"))

val firstwords = lines.map(line => (line.substring(0,line.indexOf(" ")))).collect()

http://ampcamp.berkeley.edu/wp-content/uploads/2013/02/Parallel-Programming-With-Spark-Matei-Zaharia-Strata-2013.pdf

page 14

Tuesday, April 19, 2016

1. IDE build
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IDESetup

2. Command line build
Start with the instructions @ this link:
https://dennyglee.com/2013/07/16/jump-start-onto-spark-0-7-2-and-scala-2-9-3-on-mac-osx/

Here are a few hints:

The url which has the list of the SPARK source packages is: http://spark-project.org/download/

Build instructions: https://spark.apache.org/docs/1.6.1/building-spark.html

Download: http://spark.apache.org/downloads.html

1. Download the appropriate tar file.
-Note the version number

2. cd /usr/local/Cellar/

3. Move the tar file
mv ~/Downloads/spark-1.6.1.tgz .

4. untar
tar xfv spark-1.6.1.tgz

5. cd to the spark directorycd spark-1.6.1

6. Building with build/mvnhttps://spark.apache.org/docs/1.6.1/building-spark.html#building-with-buildmvn

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

wait a while
...

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 27:38 min

[INFO] Finished at: 2016-04-19T15:54:29-07:00

[INFO] Final Memory: 406M/1569M

[INFO] ------------------------------------------------------------------------

From here, you can now run Spark examples. Just in case, run the conf/spark-env.sh to set the Scala enviornment variables.

conf/spark-env.sh
./run spark.examples.SparkLR local[2]
./run spark.examples.SparkPi local[4]

and to run the spark shell:

./spark-shell

Trouble shooting

1. sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.8.5: not found

$ grep sbt-assembly */*

project/plugins.sbt:addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.8.5")

https://github.com/sbt/sbt-assembly

$ cd
$ sbt sbtVersion


   ...

[info] 0.13.11



$ sbt sbt-version

This prints the sbt version used in your current project, or if it is a multi-module project for each module.

Resources:

http://stackoverflow.com/questions/22853151/all-sbt-plugins-not-found

https://github.com/twitter/scalding/issues/27
http://www.scala-sbt.org/release/docs/Setup.html
http://genomegeek.blogspot.com/2014/11/how-to-install-apache-spark-on-mac-os-x.html

sbt/sbt clean assembly

Sunday, April 17, 2016

OSX Emacs setup with prelude

If you want to do a clean install of Emacs 24.5 =< then you should remove the current version of emacs on your machine and mv your ~/.emacs file to ~/.emacs.bak.

You can choose to use Emacs prelude or Cask. Prelude is probably a better choice.

# Setting up emacs from scratch on OSX

mv ~/.emacs ~/.emacs.bak

# install Emacs

http://www.wikemacs.org/wiki/Installing_Emacs_on_OS_X

brew update

sudo rm /usr/bin/emacs

sudo rm -rf /usr/share/emacs

brew install emacs --with-cocoa

# install Emacs prelude

https://github.com/bbatsov/prelude

curl -L https://git.io/epre | sh