SPARK and AVRO

SBT reminder

https://spark.apache.org/docs/latest/quick-start.html

https://github.com/databricks/spark-avro

Has an example: "Avro Data Source for Apache Spark"

The examples on the wiki use an Avro file available for download here:

https://spark-packages.org/package/databricks/spark-avro

http://stackoverflow.com/questions/23944615/how-can-i-load-avros-in-spark-using-the-schema-on-board-the-avro-files

http://www.bigdatatidbits.cc/2015/01/how-to-load-some-avro-data-into-spark.html

Fix for last line:

avroRDD.map(l => { new String(l._1.datum.get("username").toString()) } ).first

https://spark.apache.org/docs/1.5.1/api/java/org/apache/spark/serializer/KryoSerializer.html

https://stackoverflow.com/questions/23944615/how-can-i-load-avros-in-spark-using-the-schema-on-board-the-avro-files

SPARK, PARQUET,

http://zenfractal.com/2013/08/21/a-powerful-big-data-trio/

https://ogirardot.wordpress.com/2015/01/09/changing-sparks-default-java-serialization-to-kryo/

KryoSerializer

http://stackoverflow.com/questions/27046178/how-can-i-read-this-avro-file-using-spark-scala

KryoRegistrator

https://gist.github.com/massie/7224868

SPARK AVRO Packages

https://spark-packages.org/package/databricks/spark-avro

MISC.

Maven deps

http://stackoverflow.com/questions/31596670/failed-to-load-class-for-data-source-com-databricks-spark-csv

    org.apache.spark
    spark-core_2.10
    1.6.1



    org.apache.spark
    spark-sql_2.10
    1.6.1



    com.databricks
    spark-csv_2.10
    1.4.0

Submitting apps to a cluster

https://spark.apache.org/docs/latest/submitting-applications.html

http://stackoverflow.com/questions/24442240/how-to-reference-jar-files-after-sbt-publish-local

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.10" % sparkVersion withSources(),
  "org.apache.spark" % "spark-sql_2.10" % sparkVersion  withSources()
)

https://github.com/sbt/sbt-assembly

add this to project/assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")

and this to build.sbt

assemblyMergeStrategy in assembly := {

  case PathList("META-INF", xs @ _*) => MergeStrategy.discard

  case x => MergeStrategy.first

}

http://stackoverflow.com/questions/34093715/scala-code-not-compiling-in-sbt

You should add to your build.sbt the following dependency:

libraryDependencies += "org.apache.spark" %% "spark-mllib" % "1.4.0"

And in your scala-file add the following import:

import org.apache.spark.{SparkConf, SparkContext}