How to install SPARK on OSX

1. IDE build
https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IDESetup

2. Command line build
Start with the instructions @ this link:
https://dennyglee.com/2013/07/16/jump-start-onto-spark-0-7-2-and-scala-2-9-3-on-mac-osx/

Here are a few hints:

The url which has the list of the SPARK source packages is: http://spark-project.org/download/

Build instructions: https://spark.apache.org/docs/1.6.1/building-spark.html

Download: http://spark.apache.org/downloads.html

1. Download the appropriate tar file.
-Note the version number

2. cd /usr/local/Cellar/

3. Move the tar file
mv ~/Downloads/spark-1.6.1.tgz .

4. untar
tar xfv spark-1.6.1.tgz

5. cd to the spark directorycd spark-1.6.1

6. Building with build/mvnhttps://spark.apache.org/docs/1.6.1/building-spark.html#building-with-buildmvn

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

wait a while
...

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 27:38 min

[INFO] Finished at: 2016-04-19T15:54:29-07:00

[INFO] Final Memory: 406M/1569M

[INFO] ------------------------------------------------------------------------

From here, you can now run Spark examples.  Just in case, run the conf/spark-env.sh to set the Scala enviornment variables.

conf/spark-env.sh
./run spark.examples.SparkLR local[2]
./run spark.examples.SparkPi local[4]

and to run the spark shell:

./spark-shell

Trouble shooting

1. sbt.ResolveException: unresolved dependency: com.eed3si9n#sbt-assembly;0.8.5: not found

$ grep sbt-assembly */*

project/plugins.sbt:addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.8.5")

https://github.com/sbt/sbt-assembly

$ cd
$ sbt sbtVersion


   ...

[info] 0.13.11



$ sbt sbt-version

This prints the sbt version used in your current project, or if it is a multi-module project for each module.

Resources:

http://stackoverflow.com/questions/22853151/all-sbt-plugins-not-found

https://github.com/twitter/scalding/issues/27
http://www.scala-sbt.org/release/docs/Setup.html
http://genomegeek.blogspot.com/2014/11/how-to-install-apache-spark-on-mac-os-x.html

sbt/sbt clean assembly 

2.